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ABSTRACT 


Telemedicine  is  a  system  of  healthcare  delivery  tools  which  uses 
telecommunications  consultations  as  an  alternative  to  transportation  of  the  patient. 
There  are  no  conclusive  studies  to  prove  or  disprove  the  use  of  telemedicine  and  it 
is  often  implemented  with  little  basis  for  measuring  its  effectiveness.  Recent 
initiatives  have  been  driven  by  advances  in  technology  and  pressure  by  upper 
management  to  reduce  the  cost  of  health  care,  but  not  from  local  needs 
assessments.  This  thesis  provides  a  methodology  to  collect  data  used  in  supporting 
measures  of  effectiveness.  The  methodology  is  developed  through  a  review  of 
strategic  goals,  an  assessment  of  potential  measures  of  effectiveness,  and  use  of  a 
model  for  data  collection.  It  is  applied  at  a  Navy  medical  treatment  facility 
recently  installing  telemedicine  equipment. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Recently,  mostly  due  to  efforts  to  reduce  health  care  costs,  there  has  been  much 

controversy  over  whether  or  not  to  investigate  the  effectiveness  of  telemedicine,  and  if  so, 

how  this  should  be  done.  Some  believe  that  telemedicine  is  already  on  its  way  to 

acceptance  and  research  is  of  secondary  importance  to  the  task  of  cultivating  clinical 

telemedicine,  while  others  believe  that  a  systematic  research  agenda  is  important  to  the 

progression  of  telemedicine  (Grigsby,  1996).  In  addition  to  the  nation’s  drive  to  reduce 

the  cost  of  managed  healthcare  and  improve  outcomes,  the  Navy’s  desire  is  to  improve 

overall  readiness  and  reduce  movement  of  the  patient.  Part  of  the  strategic  goal 

formulated  by  the  Navy’s  Bureau  of  Medicine  (BUMED)  is  to  “encourage  the  use  of 

appropriate  available  technologies  to  enhance  and  measure  quality  of  care,  readiness, 

patient  satisfaction,  resource  utilization,  and  efficiency.”  This  has  proven  difficult, 

however,  since  it  requires  the  collection  and  interpretation  of  data  that  may  not  be 

consistent  across  facilities.  Furthermore,  according  to  a  January  1997  report  to  Congress 

by  the  Department  of  Commerce: 

“Although  many  individuals  believe  strongly  in  the  potential  of 
telemedicine  for  providing  cost-effective  services,  not  much  "hard  data"  is 
available  to  support  that  belief.  Decision-makers  want  to  know  the  value- 
added  of  telemedicine.  Lack  of  solid  evaluative  information  is  a  significant 
barrier  to  the  deployment  of  telemedicine.” 

There  is  not  a  clear  methodology  to  collect  data  useful  in  making  decisions  about 
telemedicine. 

Outcomes  research  normally  provides  validation  of  effectiveness,  but  given  the 


1 


variable  nature  of  telemedicine,  the  study  of  telemedicine  outcomes  can  be  quite  complex 
and,  implementation  can  be  costly  in  terms  of  manpower  to  analyze  and  make  a 
comparison.  This  works  against  third-party  payers  who  demand  evidence  that  not  only 
health  services  work,  but  that  they  are  not  excessively  costly. 

In  this  thesis,  I  examine  the  deployment  of  a  telemedicine  suite  during  September 
1996.  I  identify  a  methodology  for  collecting  data  to  establish  measures  of  effectiveness, 
while  discussing  the  use  of  a  data  collection  model.  I  then  focus  on  the  U.S.  Naval 
Hospital,  Rota,  Spain,  as  a  case  study  for  application  of  the  model.  The  focus  of  this 
thesis  is  on  using  acquired  information  to  decide  where  to  place  further  investments  in 
telemedicine. 

To  understand  the  motives  for  this  topic,  reference  is  made  to  the  DoD  Guide  for 

Managing  Information  Technology  (IT)  as  an  Investment  and  Measuring  Performance 

which  defines  IT  performance  measurement  as: 

“the  assessment  of  effectiveness  and  efficiency  of  IT  in  support  of  the 
achievement  of  an  organization’s  missions,  goals,  and  quantitative 
objectives  through  the  application  of  outcome-based,  measurable,  and 
quantifiable  criteria,  compared  against  an  established  baseline,  to 
activities,  operations,  and  processes.” 

B.  RESEARCH  QUESTIONS 

Using  the  Naval  Hospital  Rota  as  a  case  study,  I  seek  to  answer  the  following  questions: 

1 .  How  can  measurements  be  developed  which  support  telemedicine 
effectiveness? 

2.  Based  on  these  measures  of  effectiveness,  how  can  we  use  them  to 
make  better  decisions  on  future  investments  of  telemedicine  to  result  in 
increased  use? 

C.  SCOPE  AND  LIMITATIONS 

The  scope  of  this  thesis  includes  a  review  of  telemedicine,  an  analysis  of  current 
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efforts  to  measure  the  effectiveness  of  telemedicine,  a  proposed  data  collection 
methodology,  and  a  case  study  of  the  deployment  of  telemedicine  at  Naval  Hospital, 

Rota,  Spain.  Since  this  is  a  qualitative  study  of  telemedicine  functions,  the  technology 
discussions  and  statistical  inferences  will  be  limited  and  designed  to  address  functional 
issues. 

D.  METHODOLOGY 

The  methodology  used  in  this  thesis  includes  a  literature  review,  consultation  with 

Navy  telemedicine  project  personnel,  analysis  of  proposed  measures  of  effectiveness 

(MOE’s),  a  proposal  for  a  data  collection  methodology,  and  a  case  study.  The  literature 

review  consists  of  books,  magazine  articles,  CD-ROM,  the  World  Wide  Web,  and  other 

library  information  services  describing  telemedicine  applications  and  technology. 

Consultation  was  conducted  with  Chief  Information  Officer’s  (CIO’s)  and  healthcare 

providers  at  Naval  hospitals.  A  case  study  at  the  Naval  Hospital  Rota,  Spain,  was 

conducted  to  test  the  applicability  of  the  MOEs  proposed  in  this  research. 

The  case  study  focused  on  examining  the  deployment  of  telemedicine  to  a  medical 

treatment  facility  with  its  own  specific  mission  and  requirements.  It  was  not  intended  to 

use  sophisticated  statistical  techniques  in  the  analysis  of  survey  responses,  since  the 

results  were  intended  only  to  provide  general  guidance  and  not  stringent  findings.  It  also 

was  not  intended  to  infer  that  the  methodologies  used  are  ideal  for  other  sites  or 

requirements.  As  Robert  Yin  states  in  Case  Study  Research  Design  and  Methods: 

Case  study  conclusions  are  generalizations  to  theoretical  propositions  and 
not  to  populations  or  universes....  In  this  sense  a  case  study  does  not 
represent  a  ‘sample’  and  the  investigator’s  goal  is  to  expand  and 
generalize  theories  (analytic  generalization)  and  not  to  enumerate 
frequencies  (statistical  generalization).  (Yin,  1988,  p.21) 
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E.  ORGANIZATION  OF  THESIS 


This  thesis  is  composed  of  six  chapters.  Chapter  II  is  a  general  review  of 
telemedicine  and  existing  barriers  to  implementation.  Chapter  III  contains  definitions 
and  methods  based  on  quality  assurance  and  outcomes.  Chapter  IV  proposes  MOEs  to  be 
used  in  the  case  study  and  Chapter  V  uses  a  data  collection  model  to  apply  these  MOEs  to 
the  hospital  in  Rota.  Chapter  VI  summarizes  the  thesis  and  presents  conclusions  and 
recommendations  drawn  from  the  case  study. 
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II.  TELEMEDICINE  BACKGROUND 


A.  DEFINITION  OF  TELEMEDICINE 

Health  Affairs  last  defined  telemedicine  in  1995: 

“Telemedicine  is  the  investigation,  monitoring,  and  management  of 
patients  and  the  education  of  patients  and  staff  using  systems  which  allow 
ready  access  to  expert  advice  and  patient  information  no  matter  where  the 
patient  or  the  relevant  information  is  located.” 


Another  definition  from  “Telemedicine  Today”  (Allen  &  Perednia,  1996)  is  as  follows: 

“Telemedicine  is  most  commonly  defined  as  the  provision  of  medical  care, 
services,  and  education  through  the  use  of  telecommunications 
technology.  The  technologies  used  range  from  the  mundane  (phones  and 
FAX  machines)  to  the  fantastic  (satellite-mediated  virtual  reality),  and  the 
medical  applications  range  from  transmission  of  medical  records  to  remote 
telesurgery.” 


The  GAO  Report  to  Congress  on  Telemedicine  (1997)  describes  it  in  three  levels: 

“At  the  lowest  level,  telemedicine  could  be  the  exchange  of  health  or 
medical  information  via  the  telephone  or  facsimile  (fax)  machine.  At  the 
next  level,  telemedicine  could  be  the  exchange  of  data  and  image 
information  on  a  delayed  basis.  A  third  level  could  involve  interactive 
audio-visual  consultations  between  medical  provider  and  patient  using 
high-resolution  monitors,  cameras,  and  electronic  stethoscopes.” 


Each  definition  has  a  consistent  information  technology  and  medical  theme;  however,  a 
universally  accepted  definition  is  still  a  point  of  contention.  It  is  this  impreciseness  that 
makes  research  and  measurements  more  complex. 

1.  Clinical  Applications 

Most  of  today’s  telemedicine  practice  falls  into  one  of  the  following  categories 
(Allen  &  Perednia,  1996): 
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•  Interactive-video  (IATV)  mediated  patient-physician  consultations 

•  IATV-mediated  healthcare  education 

•  Teleradiology 

•  Telepathology 

•  Miscellaneous  store-and-forward  (S&F)  applications:  teledermatology, 

tele-ophthalmology,  and  transfer  of  medical  information  on  the  Internet. 

These  functions  are  performed  with  the  use  of  peripheral  equipment  such  as  video¬ 
teleconference  (VTC)  equipment,  electronic  stethoscopes,  oto-ophthalmoscopes,  and 
close-up  cameras. 

The  distinguishing  feature  of  telemedicine  is  the  communications  link  between 
devices.  This  link  provides  bandwidth,  or  a  “pipeline,”  and  is  the  determining  factor  of 
how  much  data  can  be  transmitted  over  a  given  time  interval.  The  cost  of  bandwidth  is 
also  a  varying  factor  depending  on  the  circuit,  the  devices  connecting  them,  and  the 
switches  in  between.  Thus,  although  bandwidth  is  usually  understood  to  be  more 
expensive  as  the  capacity  increases,  sometimes  it  can  be  quite  inexpensive.  Generally  the 
IATV  applications  require  more  bandwidth  than  other  telemedicine  applications. 

Applications  which  thrive  are  those  which  are  not  typically  available  by 
traditional  means.  Some  medical  applications  are  less  sensitive  to  tactile  requirements  of 
healthcare  and  better  suited  for  remote  consultation  through  the  use  of 
telecommunications  technology.  Disciplines  and  applications  that  are  weighted  toward 
cognitive,  visual,  and  aural  information  are  more  suited  to  telemedicine.  Data,  images, 
and  speech  are  most  faithfully  transmitted  in  this  way.  The  more  dependent  an 
interaction  is  on  touch  or  smell,  the  less  likely  it  is  that  telemedicine  can  fully  substitute 
for  an  on-site  interaction.  Thus,  according  to  Allen  (1996),  clinical  applications  of 
telemedicine  today  (from  most  to  least  used)  are: 
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•  Radiology  (S&F;  can  be  done  over  regular  phone  lines) 

•  Psychiatry  (IATV;  has  been  used  with  ‘low-end’  systems  running  at  128  Kbps) 

•  Dermatology  (S&F;  IATV) 

•  Internal  medicine  specialties:  (usually  IATV) 

Cardiology 

Other  (oncology,  endocrinology,  infections  disease,  etc.) 

•  Neurology  (IATV) 

•  Home  health  care  (IATV;  over  50%  of  home  nurse  visits  do  not  require  hands- 

on  interaction 

•  Surgery  (IATV) 

Follow-ups  and  initial  assessments 

•  Emergency  room  (IATV;  links  between  small  hospitals  and  urban  E.R.s) 

•  Nursing:  (IATV) 

Counseling 

Education 

•  Pathology  (S&F;  IATV) 

•  Allied  health:  (IATV) 

Rehabilitation 
Physical  therapy 

•  Ophthalmology  (S&F;  IATV) 

•  Dentistry  (IATV) 


2.  Telemedicine  Technology 

In  order  to  provide  an  operational  telemedicine  suite,  there  are  three  basic 
technical  elements  that  must  be  in  place  and  integrated:  end  components,  standards,  and 
communications  links. 

a.  End  Components 

Components  are  being  developed  with  increasing  intensity  in  order  to 
satisfy  the  demand  of  an  increasing  set  of  applications.  The  most  commonly  used 
instruments  are  digital  stethoscopes,  digital  ECG’s,  video  otoscopes,  surgical  and  patient 
cameras,  digital  ultrasound,  dermatology  microscopes,  video  ophthalmoscopes,  and 
dental  cameras.  As  new  equipment  is  introduced,  it  must  be  evaluated  as  a  medical 
device  for  use  by  telemedicine  by  the  Food  and  Drug  Administration  (FDA).  The 
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Medical  Device  Amendments  of  1976  (Public  Law  94-295)  and  the  Safe  Medical  Device 
Amendments  of  1990  (Public  Law  101-629)  established  a  comprehensive  system  for  the 
regulation  of  medical  devices  intended  for  human  use.  This  can  sometimes  prove  time-  • 
consuming  since  lack  of  classification  regulations  has  not  allowed  the  FDA  to  establish 
exemption  from  any  of  the  general  controls  for  some  of  the  devices  where  it  would  be 
appropriate.  For  example  the  use  of  a  personal  computer  to  view  medical  images  could 
be  classified  as  a  "medical  device"  and  subject  to  FDA  approval.  To  remedy  this 
situation  a  classification  proposal  has  been  developed  and  is  awaiting  Federal  Register 
publication  establishing  classifications  for  five  medical  image  management  devices 
(FDA,  1996). 

b.  Standards 

Standards  are  being  sought  for  all  aspects  of  telemedicine  networks,  but 
few  have  been  implemented  with  the  exception  of  the  American  College  of  Radiology 
Teleradiology  Standard  (United  States  Department  of  Commerce,  1997)  which  includes 
equipment  guidelines.  In  order  for  a  telemedicine  system  to  work,  the  standards  that 
must  be  in  place  are  communications  standards,  file  compression  standards,  and 
equipment  standards.  These  standards  offer  a  more  reliable  and  robust  link,  and  a 
standard  applied  consistently  across  multiple  activities  offers  ensured  connectivity  with 
more  sites.  Agencies  with  active  roles  in  producing  acceptable  standards  for  medical 
equipment  and  telecommunications  include  the  Food  and  Drug  Administration  (FDA), 
the  Federal  Communications  Commission  (FCC)  Advisory  Committee  on 
Telecommunications  and  Health  Care,  and  the  National  Institute  of  Standards  and 
Technology  (NIST).  Within  the  FDA,  the  Center  for  Devices  and  Radiological  Health 
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recognizes  that  the  proliferation  of  “wireless”  communications  that  can  link  the  devices 
together  and  aid  in  the  diagnosis  and  treatment  of  patients  has  introduced  electromagnetic 
interference  (EMI)  with  medical  devices  and  has  become  a  source  of  concern. 

Interference  can  cause  problems  with  other  communications  devices  or  devices  dependent 
on  reception  signals.  Direct  wired  communications  devices  can  suffer  from  EMI.  The 
impact  of  these  technologies  must  be  addressed  in  the  formation  of  standards  (FDA, 
1996). 

c.  Communications  links 

Telecommunications  is  the  most  difficult  technology  to  plan  for  and 
maintain,  since  telecommunications  technologies  are  evolving  at  such  a  rapid  rate. 
Equipment  and  protocols  must  be  compatible  and  integratable  to  connectable  sites. 
Planning  is  further  complicated  by  acquisition  regulations,  financial  costs,  and  anticipated 
use  of  the  link.  Following  are  some  of  the  technologies  to  be  considered  when  planning 
telemedicine  links. 

The  Public  Switched  Telephone  Network  (PSTN)  is  the  most  commonly 
accessible.  It  is  a  “common  carrier”  and  available  to  anyone  or  any  organization  who  can 
afford  to  use  it.  Most  of  the  services  using  PSTN  are  traditional  voice,  facsimile,  or  data 
transmissions.  Since  PSTN  was  originally  designed  for  voice  transmission,  it  is  limited 
in  speed  and  quality,  thus  high  quality  VTC  and  high  resolution  image  files  are  not 
always  possible.  A  2400  baud  rate  is  typical,  but  compression  techniques  can  improve 
the  rate  to  56  Kbps. 

Another  option  is  the  Integrated  Systems  Digital  Network  (ISDN).  In  this 
network,  all  traffic  is  digitally  encoded  allowing  use  of  more  intelligent  services  using 
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compression  and  allowing  greater  capacity.  The  main  advantage  is  that  several  services 
can  be  operated  simultaneously  from  one  line.  For  example,  two  researchers  could  be 
looking  at  separate  views  of  the  same  file  being  transmitted,  while  talking  on  the  phone 
or  receiving  a  fax.  Although  the  same  PSTN  line  can  be  used,  new  switches  must  be 
installed  to  accommodate  the  digital  requirement  and  thus  may  be  expensive  to 
implement. 

The  Broadband  Switched  Network  acts  much  like  a  multimedia 
workstation  in  which  a  physician  receives  many  types  of  interactive  portions  of  a  consult. 
By  definition,  broadband  offers  higher  than  a  1.54  Mbps  rate;  however,  the  network 
would  have  to  be  rebuilt  for  switched  broadband  with  new  wiring,  connections,  and 
signaling  algorithms. 

The  Internet  is  the  newest  and  fastest  growing  application  of  networking 
and  is  being  integrated  not  only  into  medicine  but  also  into  everyday  life.  Some 
telemedicine  applications  are  already  being  implemented  on  the  Internet.  The  Internet 
grew  out  of  a  requirement  by  the  Advanced  Research  Projects  Agency  (ARPA)  for  shared 
information  among  computers.  Using  the  Transmission  Control  Protocol/Intemet 
Protocol  (TCP/IP)  it  has  evolved  into  a  network  of  networks  and  an  inexpensive  way  to 
send,  receive,  and  view  information  in  the  form  of  text  or  graphics.  It  is  now  also 
possible  to  use  interactive  audio  over  the  Internet.  An  example  of  the  benefits  of  the 
Internet  is  a  project  with  the  South  Texas  Health  Research  Center  (STHRC).  STHRC 
originally  desired  to  link  all  the  hospitals,  clinics,  and  training  sites  in  South  Texas  in  a 
sophisticated  telecommunications  network.  This,  however,  would  have  proven  to  be  cost 
prohibitive  so  they  designed  a  shortcut  to  (1)  create  computer-stored  files  of  the  STHRC 
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information  often  requested  out  in  the  field,  (2)  install  those  files  in  an  STHRC  “file 
server”  (essentially  a  central  computer),  (3)  connect  that  STHRC  server  to  the  Internet 
with  appropriate  menus  and  file  indexes,  (4)  encourage  local  clinics  and  other  sites  to  link 
their  desktop  computers  via  local  phone  lines  or  state  telecommunications  lines  to  the 
nearest  Internet  access  point,  and  (5)  develop  training  so  individual  users  in  the  field  will 
know  the  few  simple  steps  for  connecting  to  STHRC  via  the  Internet  to  request  needed 
information  or  to  exchange  e-mail.”  Even  though  bells  and  whistles  were  not 
implemented  with  the  solution,  the  functionality  of  a  telemedicine  service  exists,  and  as 
usage  increases  or  decreases  a  more  appropriate  solution  can  be  achieved  incrementally. 

Another  evolving  link  is  through  the  use  of  digital  radio.  This  provides 
greater  portability  of  terminals, -services,  and  integration  of  broadcast  and  wired  networks. 
Radio  is  not  currently  used  as  expansively  as  hard  wire,  although  it  could  be  a  viable 
alternative  in  consultations  occurring  in  extremely  remote  areas. 

B.  HISTORY 

Moore  (1995)  dates  telemedicine  back  to  the  1920s,  when  radio  was  used  to  link 
public  health  physicians  standing  watch  at  shore  stations  in  order  to  assist  ships  at  sea  that 
had  medical  emergencies.  However,  one  of  the  first  organized  developments  of 
telemedicine  was  that  by  NASA  in  the  early  1960s  when  humans  began  flying  in  space. 
Physiological  parameters  were  monitored  from  both  the  spacecraft  and  the  space  suits 
during  missions  (Bashshur,  1975). 

One  of  the  earliest  deliveries  of  medical  care  with  the  use  of  telemedicine  was 
also  conceived  by  NASA.  The  Space  Technology  Applied  to  Rural  Papago  Advanced 
Health  Care  (STARPAHC)  delivered  medical  care  to  the  Papago  Indian  Reservation  in 


11 


Arizona  from  1972  to  1975  while  also  providing  health  care  to  astronauts  in  space.  The 
instruments  included  an  electrocardiograph  and  x-ray  and  were  linked  with  a  two-way 
microwave  telemedicine  and  audio  transmission  (Bashshur,  1980). 

The  Nebraska  Psychiatric  Institute  was  one  of  the  first  facilities  in  the  country  to 
have  closed-circuit  television  in  1955.  The  link  was  with  Norfolk  State  Hospital  which 
was  112  miles  away  and  was  used  for  consultations,  education,  and  later  experimentation 
with  group  therapy  (Benschoter,  1971). 

There  have  been  numerous  other  miscellaneous  applications  of  telemedicine.  In 
1967,  Massachusetts  General  Hospital  and  Logan  International  Airport  Medical  Station 
occupational  health  services  were  provided  to  airport  employees  and  emergency  care  was 
provided  to  travelers  (Murphy,  1973).  In  1971, 26  sites  in  Alaska  were  chosen  by  the 
National  Library  of  Medicine  to  see  if  reliable  communication  would  improve  village 
health  care  (Foote,  1976).  The  first  international  telemedicine  program  was  conducted  by 
NASA  in  1989  as  a  result  of  a  massive  earthquake  in  the  Soviet  Republic  of  Armenia. 
Under  the  auspices  of  the  U.S./U.S.S.R.  Joint  Working  Group  on  Space  Biology, 
telemedicine  consultations  were  conducted  using  one-way,  voice,  and  facsimile  between 
a  medical  center  in  Yerevan,  Armenia  and  four  medical  centers  in  the  U.S.  This  project 
demonstrated  that  medical  consultation  could  be  conducted  over  a  satellite  network 
crossing  political,  cultural,  social,  and  economic  borders.  Even  more  recently,  NASA’s 
Lewis  Research  Center  in  Cleveland,  OH  worked  with  NASA's  Ames  Research  Center  in 
Moffett  Field,  CA  to  provide  echocardiographic  examinations  to  be  remotely  diagnosed 
by  a  doctor  at  the  Cleveland  Clinic.  It  is  hoped  that  this  will  pave  the  way  for  physicians 
on  Earth  to  view  the  heart  function  of  an  astronaut  aboard  the  International  Space  Station 
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(Nolan-Proxmire,  1997). 

Even  though  telemedicine  has  existed  for  decades,  the  official  start  of 
telemedicine  in  DoD  is  September  29, 1994,  when  the  U.S.  Army  Medical  Research  and 
Materiel  Command  published  a  draft  Telemedicine  Testbed  Master  Plan.  In  January 
1995,  it  published  a  draft  Telemedicine  Strategic  Plan.  Since  then,  initiatives  have  been 
ongoing  to  include  validation  initiatives  at  the  Naval  Health  Research  Center  (NHRC). 
Other  leading  DoD  organizations  involved  in  formulating  national  and  DoD  telemedicine 
policy  initiatives  include  the  DoD  Advanced  Research  Projects  Agency,  the  Office  of  the 
ASD(HA),  and  the  U.S.  Army  Medical  Research  and  Materiel  Command.  The  ASD(HA) 
has  established  a  DoD  Telemedicine  Testbed  Board  of  Directors  which  supports  national 
information  infrastructure  initiatives  and  is  the  steering  group  for  DoD  telemedicine 
development  and  utilization  initiatives.  However,  even  though  collaboration  has  begun  to 
establish  a  standard  methodology  for  assessing  the  effectiveness  of  telemedicine,  it  has 
yet  to  be  accepted  and  supported  service-wide,  and  local  sites  are  left  with  the  task  of 
setting  strategic  goals  for  the  implementation  of  telemedicine. 

C.  EXISTING  BARRIERS 

Looking  past  the  lack  of  project  evaluation,  there  are  still  consistent  and 
recognizable  barriers  to  realizing  the  full  benefits  of  telemedicine.  GAO  (1997)  has 
acknowledged  that  these  barriers  can  be  categorized  as  legal  and  regulatory,  financial, , 
technical,  and  cultural.  They  all  have  a  basis  for  deterrent,  and  when  one  barrier  is 
overcome,  the  other  three  can  still  cause  a  particular  program  to  fail.  The  advantage  for 
the  Federal  Government  is  that  it  is  less  affected  by  legal  and  regulatory  barriers  due  to 
the  Federal  Tort  Claims  Act  (FTCA).  This  is  why  most  organizations  are  looking  to  the 
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Federal  government  to  investigate  new  protocols  and  technologies  and  to  conduct  studies. 

1.  Legal  Barriers 

The  most  significant  barrier  in  private  telemedicine  practice  is  the  requirement  for 
health  care  professionals  to  be  credentialed  in  the  state  in  which  care  is  provided.  Some 
states  prohibit  care  provided  through  telemedicine  (GAO,  1997).  Although  special 
licenses  are  available,  they  are  not  always  accepted  by  other  states.  For  federally 
employed  professionals  this  is  not  an  issue,  since  they  must  simply  be  licensed  in  one 
state  in  order  to  provide  care  to  a  patient  eligible  for  federal  benefits.  However,  this 
affects  DoD  healthcare  if  a  civilian  physician  is  prohibited  by  state  law  to  practice 
telemedicine.  There  are  a  variety  of  models  under  review  that  could  be  applied  to  a 
health  care  professional  such  as  consulting  exceptions,  endorsement,  mutual  recognition, 
reciprocity,  registration,  limited  licensure,  national  licensure,  and  federal  licensure. 

These  models  are  a  long  way  from  approval  by  all  states,  however.  This  is  an  area  where 
the  Federal  government  might  serve  as  a  model  for  states  to  follow. 

Malpractice  is  also  an  issue  for  every  doctor-patient  encounter.  Currently  a 
physician  is  held  responsible  for  the  treatment  given  to  patients.  However,  if  the  consult 
is  by  telemedicine  it  is  difficult  to  determine  in  which  geographical  jurisdiction  the  suit 
should  be  addressed.  With  federal  employees,  the  controlling  law  is  the  Federal  Tort 
Claims  Act  (FTCA),  which  for  more  than  forty  years  “has  been  the  legal  mechanism  for 
compensating  persons  injured  by  negligent  or  wrongful  acts  of  Federal  employees 
committed  within  the  scope  of  their  employment.”  FTCA  prevents  federal  employees 
from  being,  sued-rather,  it  is  an  action  against  the  United  States.  As  of  the  1997  GAO 
report,  only  4  known  suits  had  been  filed  against  private  physicians,  and  no  suits  had 
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been  filed  against  the  federal  government. 

The  privacy  and  security  of  medical  data  must  also  be  addressed.  For  example, 
those  records  that  are  considered  classified  (as  in  patients  with  AIDS)  are  at  increasing 
risk  when  being  transferred  over  public  communications  links.  This  is  compounded  by  a 
failure  for  facilities  to  have  policies  and  procedures  in  place  to  address  cases  that  could  be 
construed  as  sensitive.  Within  DoD,  medical  data  is  protected  in  accordance  with  the 
Privacy  Act  of  1974  and  the  DoD  AIS  Security  Program  which  mandates  the  employment 
of  technical  and  physical  safeguards  to  ensure  the  information  is  adequately  protected  by 
health  care  providers  and  administrators. 

2.  Financial  Barriers 

Difficulties  in  determining  reimbursements  have  deterred  the  expansion  of 
telemedicine  services.  Often  a  consulting  physician  is  not  given  monetary  or  workload 
credit  for  performing  a  tele-consult.  In  addition.  Medicare  does  not  pay  for  telemedicine 
because  it  believes  the  standard  practice  of  medicine  requires  an  “in-person,  face-to-face 
consultation”  between  the  patient  and  practitioner  for  those  consultations  requiring 
dialogue.  However,  in  October  1996,  HCFA  announced  that  it  will  begin  limited 
Medicare  payments  for  telemedicine  consultation  in  four  states  under  a  demonstration 
project  (GAO,  1997).  In  a  managed  care  organization  increased  access  may  result  in 
increased  utilization  and  thus  increased  cost.  However,  expanded  use  of  capitated 
managed  care  systems  should  enhance  the  appeal  of  telemedicine  and  reduce  the  need  for 
reimbursement. 

The  financial  barrier  to  smaller  hospitals  or  clinics  is  the  cost  of  implementing  a 
telemedicine  infrastructure  which  can  range  from  several  thousand  to  hundreds  of 
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thousands  of  dollars.  Even  though  the  benefits  of  a  high-bandwidth  connection  from  a 
small  rural  hospital  to  a  major  health  care  center  are  high,  smaller  facilities  may  be  the 
least  able  to  pay  for  these  services.  In  these  cases,  resource  planners  must  be  able  to 
leverage  additional  services  with  the  infrastructure  installation,  such  as  education  or 
entertainment. 

3.  Technical  Barriers 

Technical  standards  are  slow  to  be  adopted,  such  as  data  definition,  coding,  and 
transmission  of  images.  These  distinctions  are  very  important  to  the  payer  and  provider, 
since  they  can  affect  which  insurance  company  will  be  liable  for  a  claim.  Much  of  the 
information  is  left  to  the  interpretation  of  individual  managed  care  organizations,  and 
providers  must  also  make  assumptions  when  coding  claim  data  elements  and  frequently 
use  coding  standards  employed  by  the  provider’s  system.  To  date,  the  only  specialty¬ 
generated  clinical  guidelines  existing  are  the  practice  guidelines  for  teleradiology 
developed  by  the  American  College  of  Radiology  (United  States  Department  of 
Commerce,  1997). 

Lack  of  technical  standards  can  result  in  equipment  and  systems  that  do  not 
communicate  with  other  equipment  and  systems  or  does  not  provide  adequate  images  for 
clinical  decision-making.  All  too  often,  lack  of  objective  technical  advice  has  led  to 
inappropriate  purchases  and  poorly  performing  systems  that  hinder  the  cost-effective  , 
application  of  telemedicine  technologies.  Equipment  is  still  installed  with  proprietary 
and  “closed”  standards  in  which  the  equipment  may  only  communicate  with  other  suites 
of  its  kind.  This  is  especially  a  problem  in  DoD,  where  portability  and  movement  is 
always  an  issue— the  location  of  the  patient  or  the  needed  specialty  could  be  anywhere  in 
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the  world. 


4.  Cultural  Barriers 

Cultural  acceptance  usually  falls  into  two  categories:  physician  acceptance  (to  • 
include  willingness  to  use  the  equipment  and  their  skepticism  about  diagnosing  and 
treating  patients  without  a  face-to-face  encounter)  and  patient  satisfaction. 

Use  by  conservative  physicians  is  dependent  upon  their  desire  to  provide  quality, 
controlled,  convenient,  and  low-risk  care  to  the  patient.  Given  the  lack  of  telemedicine 
research,  there  is  a  substantial  level  of  resistance  to  accept  the  process  of  telemedicine 
consultation.  Convincing  physicians  that  telemedicine  can  provide  appropriate,  if  not 
better,  level  of  care  requires  significant  training,  discussion  with  peers,  and 
experimentation  by  the  physician.  Downplay  of  physician  participation  has  been 
documented  such  as  in  a  1995  journal  article  by  the  Health  and  Human  Services  (HHS) 
and  the  Telemedicine  Center,  Medical  College  of  Georgia,  in  which  it  is  stated  that  the 
designs  of  current  systems  are  driven  more  by  technology  than  by  the  needs  of  physicians 
(Puskin  and  Sanders,  1995).  In  these  cases,  telemedicine  technologies  must  adapt  to  the 
needs  of  users,  not  vice  versa,  in  order  to  optimize  the  buy-in  of  healthcare  providers. 
Training  was  cited  by  Puskin  and  Sanders  as  a  key  component  of  any  successful 
telemedicine  system. 

Patient  satisfaction,  according  to  GAO,  may  be  less  of  a  barrier  than  physician 
acceptance,  particularly  in  rural  settings.  As  compared  with  driving  hundreds  of  miles  for 
health  care,  telemedicine  is  a  much  more  appealing  option  than  seeing  a  specialist  face- 
to-face.  Projects  in  South  Dakota  and  Florida  found  approximately  70  percent  of  the 
patients  preferred  telemedicine  consultations  and  14  percent  were  neutral.  Another 
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project  in  Kansas  involved  patients  who  were  content  avoiding  a  300  or  400  mile  drive; 
however,  they  found  it  difficult  to  be  candid  on  video  and  were  not  eager  to  repeat  their 
experiences  (GAO,  1997).  In  DoD,  however,  it  is  difficult  to  measure  patient  satisfaction 
in  an  overseas  setting,  since  it  is  often  a  benefit  in  the  patient’s  eyes  to  be  transported  to  a 
major  hospital  for  the  purpose  of  improved  morale.  Patients  are  more  “satisfied”  if  the 
location  of  treatment  is  “home.”  This  is  where  cultural  differences  surface  regarding 
civilian  and  military  settings  which  affect  the  perception  of  telemedicine. 
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m.  MEASURES  OF  EFFECTIVENESS  AND  BACKGROUND 
A.  DEFINITIONS 

There  are  many  different  ways  to  measure  effectiveness  of  clinical  information 
systems.  The  purpose  of  measures  of  effectiveness  is  to  discern  the  extent  to  which  an 
objective  is  achieved.  The  Joint  Medical  Operations-Telemedicine  Advanced  Concepts 
Technology  Demonstration  (JMO-T  ACTD)  defines  an  MOE  as  a  "quantifiable  measure 
used  in  comparing  systems  or  concepts  or  estimating  the  contribution  of  a  system  or 
concept  to  the  effectiveness  of  a  military  force.  It  expresses  the  extent  to  which  a  combat 
system  accomplishes  or  supports  a  military  mission."  There  are  also  measures  of 
performance  which  help  assess  effectiveness;  these  are  usually  more  quantifiable  points 
such  as  response  time,  exceptions,  and  errors  derived  from  a  piece  of  equipment  or  an 
information  system.  The  JMO-T  ACTD  defines  a  measure  of  performance  (MOP)  as  an 
MOE  "that  expresses  the  extent  to  which  a  combat  system  accomplishes  a  specific 
performance  function.  In  general,  higher-level  MOPs  are  themselves  composed  of  either 
lower-level  MOPs  or  data  requirements."  A  system  can  be  considered  effective  if  it 
meets  its  performance  requirements  or  it  contributes  in  a  significant  way  to  fulfilling  long 
term  strategic  needs  within  an  allocated  budget  (Whatmore,  1995).  Measures  of 
performance  are  concerned  with  whether  a  system  is  “doing  it  right.”  Measures  of 
effectiveness,  however,  are  concerned  with  “doing  the  right  thing.”  Thus,  if  an  objective 
is  for  a  system  to  provide  optimal  readiness,  measures  of  effectiveness  should  measure 
achievement  of  that  objective. 

Descriptions  of  effectiveness  measures  in  the  literature  are  typically  conflicting  or 
vague.  There  are  many  definitions  of  efficiency,  effectiveness,  cost/benefit  analyses, 
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performance  measurements,  goals,  outputs,  and  objectives  and  the  terms  are  sometimes 
used  interchangeably.  It  is  also  common  to  find  generalizations  with  little  tangible 
support.  Healthcare  is  a  challenge  for  managers  wishing  to  collect  measures  of 
effectiveness  since  the  objectives  are  difficult  to  define,  as  is  the  definition  of  the  quality 
of  health  care.  The  health  care  industry  is  laden  with  intangible  costs  and  benefits,  and  it 
is  this  ambiguity  that  makes  data  difficult  to  gather  and  interpret.  It  is  not  unusual  or 
necessarily  inappropriate  to  find  disagreement  in  deciding  what  to  measure.  It  has  been 
noted  (Connelly,  1993)  that  the  method  for  generating  measures  of  effectiveness  rests  on 
the  principal  that  an  MOE  is  “acceptable  to  an  individual  only  if  it  rates  effectiveness  in  a 
manner  consistent  with  the  ratings  of  that  individual  whose  judgments  are  important  for 
the  area  of  interest.” 

B.  METHODS  FOR  DEFINING  MOE'S 

In  research,  measurement  consists  of  assigning  numbers  to  empirical  events  in 
compliance  with  a  set  of  rules  (Cooper  &  Emory,  1995),  but  it  is  not  easy  to  measure 
properties  like  satisfaction-one  must  infer  its  presence  or  absence  by  observing  some 
indicant  or  pointer  measurement.  Nonetheless,  measures  must  start  somewhere  and 
objectives,  or  goals  that  are  being  sought  or  maintained,  are  the  foundation. 
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Figure  1.  Hierarchy  of  Objectives  and  Effectiveness 
(Whatmore,  1995) 

There  is  a  hierarchy  of  objectives  which  naturally  evolve  into  a  corresponding 
hierarchy  of  effectiveness  as  shown  in  Figure  1  (Whatmore,  1995).  Objectives  define  the 
requirements  for  lower  levels  until  a  performance  measure  is  identified  to  support  the 
measurement  of  effectiveness  of  the  objectives.  This  is  a  good  illustration  of  the  process 
that  occurs  within  the  monitoring  of  systems.  It  also  helps  in  isolating  separate  factors 
for  integrated  systems  in  which,  for  one  strategic  objective,  there  are  several  mission 
objectives. 

Another  approach  published  by  Wakulc2yk  describes  a  process  that  he  states  is  a 
low-risk,  methodic  approach  to  selecting  metrics  for  a  metrics  program.  He  states  that 
there  is  a  lack  of  software  managers  of  practitioners  who  have  experience  applying 
influential  references  (most  specifically  the  Malcolm  Baldridge  Award  criteria  and  the 
Software  Engineering  Institute  Capability  Maturity  Model  for  software).  This  implies 
that  some  organizations  are  inadvertently  creating  metrics  programs  without  the  required 


21 


foundation  of  organizational  definition.  Wakulczyk’s  method  describes  a  three- 
component  approach  that  considers  organizational  definition  to  be  a  prerequisite  to  truly 
successful  metrics:  analyze  the  present,  envision  the  future,  THEN  create  metrics  that 
will  allow  the  organization  to  become  its  envisioned  state.  This  is  similar  to  the  method 
illustrated  by  Whatmore,  who  attempts  to  identify  objectives  (strategic,  mission, 
equipment)  before  identifying  measures  of  effectiveness.  It  is  the  attempted  use  of  these 
approaches  by  analyzing  the  present  and  future  states  and  identifying  the  most  apparent 
and  visible  measures  based  on  the  convergence  of  relevant  data  that  is  available. 

Once  objectives  have  been  identified,  associated  measures  must  be  identified  and 
defined,  keeping  in  mind  that  a  good  measurement  tool  must  be  easy  to  use,  and  meet  the 
criteria  of  validity,  reliability,  and  practicality: 

“Validity  refers  to  the  extent  to  which  a  test  measures  what  we 
actually  wish  to  measure.  Reliability  has  to  do  with  the  accuracy  and 
precision  of  a  measurement  procedure...  Practicality  is  concerned  with  a 
wide  range  of  factors  of  economy,  convenience,  and  interpretability.” 

(Thorndike  &  Hagen,  1977) 

C.  QUALITY  ASSURANCE  AND  OUTCOME  MEASUREMENTS 

A  common  mistake  is  that  an  effort  to  identify  appropriate  measures  exceeds  the 
ability  of  the  manager  to  collect  the  data.  The  data  is  then  incorrectly  manipulated  until  it 
is  perceived  that  validity,  reliability,  or  practicality  has  been  met. 

1.  Quality  Assurance  through  Statistical  Data 

The  main  vehicle  by  which  health  care  is  monitored  is  through  a  quality  assurance 
program  (sometimes  called  performance  improvement).  A  good  quality  assurance 
program  requires  valid,  reliable  data.  As  defined  by  Sink  (1985),  quality  is  “the  degree  to 
which  the  system  conforms  to  requirements,  specifications,  or  expectations.”  Quality 
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assurance  exists  in  health  care  to  assure  beneficiaries  that  they  are  getting  the  service  they 
are  entitled  and  expect  to  receive.  Currently,  the  Joint  Commission  on  Accreditation  of 
Healthcare  Organizations  (JCAHO)  is  the  leading  organization  responsible  for 
conducting  accreditation.  According  to  standards  set  by  JCAHO  in  1995,  health  care 
organizations  are  expected  to  use  aggregate  data  and  information  to  examine  trends  over 
time,  make  comparisons  within  the  organization  and  among  organizations,  and  utilize 
management  information  obtained  to  improve  performance.  Quality  assurance  programs 
use  a  variety  of  benchmarks  and  standards  (such  as  inpatient  length  of  stay)  which  are 
typically  published  by  JCAHO  or  BUMED,  but  individual  sites  are  given  latitude  to 
implement  their  own  overall  program. 

2.  Outcome  Measurements 

Measuring  outcomes  represents  a  new  strategic  direction  for  managing  health  care 
experiencew  for  beneficiaries.  It  presents  an  opportunity  to  enhance  patient  care  and 
decrease  health  care  costs  through  stages  of  identification,  intervention,  and  monitoring 
and  evaluation.  If  collected  and  used  appropriately,  outcome  data  can  be  used  to  improve 
clinical  efficiency  and  quality  and  feed  that  information  back  into  a  quality  assurance  and 
quality  improvement  programs  to  improve  patients’  health  care.  Furthermore,  outcomes 
can  identify  areas  which  require  attention  from  senior  managers  (Kretz,  1996).  Simply 
stated,  if  outcomes  can  be  identified  with  a  new  process,  a  change  in  outcomes  will  affect 
decisions  to  alter  or  enhance  the  process. 

A  planned  change  always  results  in  resistance,  thus  a  new  way  of  practicing 
medicine  results  in  careful  scrutiny  of  changes  in  outcomes.  Providers  are  under  intense 
pressure  to  improve  the  quality  of  care,  and  introducing  telemedicine  has  introduced  risk 


23 


to  the  consistency  and  improvement  of  care.  However,  as  an  impetus  to  change, 

Congress  is  increasing  pressure  to  evaluate  telemedicine  programs  and  to  identify 
coordination  among  services.  The  Joint  Working  Group  on  Telemedicine  (JWGT)  has 
constructed  a  Federal  evaluation  framework  made  up  of  six  areas,  one  of  which  is  clinical 
outcomes.  The  question  posed  by  GAO  is,  “Are  acceptable  outcomes  associated  with  the 
use  of  telemedicine?”  and  hospitals  must  be  prepared  to  justify  the  answer.  GAO, 
however,  has  acknowledged  that  JWGT  efforts  at  coordination  among  services  have  been 
hampered  by  several  factors  including  lack  of  a  consistent  definition  and  incompatible 
agency  data.  A  definition  of  outcomes  from  Avedis  (1989)  enforces  this:  “Outcomes  are 
those  changes,  either  favorable  or  adverse,  in  the  actual  or  potential  health  status  of 
persons,  groups,  or  communities,  that  can  be  attributed  to  prior  or  current  care.”  Given 
this  definition,  few  organizations  are  ready  to  deliver  outcomes  for  3  reasons,  according 
to  Kretz: 

1.  The  industry  is  segmented  among  local  providers  of  care,  mid-level 
companies  that  specialize  in  delivering  one  service,  and  large  national 
companies. 

2.  Outcomes  analysts  find  it  difficult  and  in  some  cases  impossible  to 
distinguish  outcomes  resulting  from  care  decisions  made  by  physicians 
from  those  made  by  home  care  providers  because  home  care  providers 
operate  under  the  orders  of  the  prescribing  physician. 

3.  The  largest  home  care  companies  have  the  staff  and  financial  resources 
needed  to  develop  outcomes  systems  but  face  significant  problems 
integrating  their  data  systems  so  that  they  can  collect  and  report  outcomes 
data. 

The  concept  of  outcomes-based  quality  improvement  is  simple  to  understand,  but 
its  implementation  is  not.  The  basic  idea  is  to  measure  patients’  clinical  status  at  two  or 
more  time  points  while  keeping  track  of  the  kinds  and  intensity  of  care  they  receive.  The 
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next  step  is  to  analyze  the  data  to  determine  which  processes  are  associated  with  good 
outcomes,  which  are  associated  with  bad  outcomes  and  modify  practices  (if  necessary)  in 
light  of  the  results.  In  the  context  of  clinical  or  health  services  research,  an  outcome  is 
the  difference  between  ratings  during  or  following  treatment.  The  effect  of  the  treatment 
is  the  difference  between  the  actual  outcome  and  the  expected  outcome  given  no 
treatment.  The  problems  associated  with  this  method  of  outcome-based  measurements  are 
best  illustrated  by  Grigsby  (1996): 

“One  problem  is  that  all  illnesses  do  not  have  the  same  course.... 

From  one  disease  to  another  there  is  significant  variability  in  what  is 
considered  favorable:  stabilization,  improvement,  or  slowing  of 
progression  may  be  good  outcomes,  depending  on  the  disorder.  Likewise, 
the  metric  may  be  quite  different:  improvement  in  activities  of  daily 
living,  decrease  in  chronic  pain,  or  resolution  of  dermatologic  symptoms. 

One  solution  may  involve  stratification  of  patients  into  categories.  You 
might  examine  all  diabetics,  or  all  Type  II  diabetics,  or  all  persons  with 
progressive  neurodegenerative  diseases,  or  persons  with  fractures  of  the 
lower  extremities.  Depending  on  your  focus,  outcome  measures  may  be 
disease-specific,  or  global.” 

In  short,  Grigsby  clarifies  what  is  obvious  to  health  care  providers,  but  harder  to  see  from 
the  information  systems  manager’s  point  of  view-that  in  order  to  evaluate  outcomes,  you 
have  to  remain  open  to  the  variations  in  disorders.  In  other  words,  when  evaluating  an 
outcome  of  a  dermatology  case  versus  an  obstetrics  case,  “it  all  depends.”  He  goes  on 
further  to  look  at  specific  cases  within  a  condition. 

“A  second  difficulty  is  that  not  all  patients  with  a  given  condition 
are  alike.  This  is  where  risk-adjustment  becomes  important.  Assume  you 
are  a  geriatrician,  and  you  want  to  study  the  outcomes  of  telemedicine  care 
for  a  sample  of  geriatric  outpatients.  Older  persons  with  serious  illnesses 
are  an  alphabet  soup  of  comorbid  conditions  are  liable  to  have  worse 
outcomes,  regardless  of  the  care  they  receive,  than  are  healthier  persons.  It 
may  be  that  they  also  are  more  likely  to  receive  consultative  services  via 
telemedicine  because  of  the  complexity  of  their  condition.  If  they  are 
compared,  without  some  kind  of  risk-adjustment  (by  stratification  or 
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statistical  means),  with  individuals  who  are  less  ill  and/or  have  a  lower 
burden  of  comorbid  illness,  you  may  incorrectly  demonstrate  that 
telemedicine  outcomes  are  worse  than  those  of  conventional  care.” 

Morrissey  (1997)  summarizes  another  view,  that  outcomes  can  be  measured  according  to 

clinical  results,  patient  satisfaction,  difference  in  functional  ability  and  the  costs  of  doing 

a  procedure,  but  “process  is  the  dial  that  you  can  adjust  to  affect  the  other  four.  If  you 

improve  the  process,  everything  else  will  get  better.”  This  idea  appears  to  support  the 

notion  that  although  possible,  outcomes  is  of  secondary  importance  and  can  tend  to  be 

reactive  rather  than  proactive.  It  does  not  solve  the  problem  that  most  sites  are  facing- 

they  can’t  provide  evidence  that  telemedicine  is  making  things  better.  This  is  why 

measures  of  effectiveness  are  necessary. 
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IV.  CRITICAL  ASSESSMENT  OF  POTENTIAL  TELEMEDICINE  MOE’S 


“With  the  exception  of  image-oriented  subspecialties,  such  as  teleradiology  and 
telepathology,  few  clinical  studies  have  documented  the  accuracy,  reliability,  or  clinical 
utility  of  most  applications  of  telemedicine  as  a  primary  diagnostic  or  therapeutic 
modality”  (Perednia,  1995).  In  general,  most  projects  and  programs  have  simply  not 
generated  enough  telemedicine  consultations  by  separate  clinical  procedures  to  make 
definite  conclusions  about  the  effectiveness  of  telemedicine,  typically  experiencing  one 
of  the  previously  discussed  barriers  that  prevent  it  from  growing.  There  are  random 
studies,  but  they  have  yet  to  produce  accepted  measures.  It  is  rarely  disputed  that  the 
technology  works,  but  the  effectiveness  can  be  disputed;  the  problems  lie  in  answering 
questions  concerning  the  disposition  of  the  technology. 

A.  STEPS  TO  IDENTIFY  MOE’S 

Telemedicine  is  a  system  composed  of  many  different  stimuli  which  are  not 
always  tangible  and  often  result  in  integrated  performance  measures.  So,  to  clarify,  DoD 
seeks  to  manage  information  technology  as  an  investment  (with  emphasis  upon  the 
methodology  of  selection,  control,  and  evaluation)  and  to  be  objective  and  outcome- 
oriented.  Two  objectives  of  Navy  Medicine  is  to  reduce  the  number  of  medical 
evacuations  and  to  optimize  the  time  spent  by  specialists  giving  care  (Greenauer,  1995). 
In  addition,  “Goal  3:  Technology”  of  the  BUMED  Strategic  Plan  states  that  the  “Navy 
Medical  Department  will  be  a  leader  in  technology  integration.”  Strategies  that  support 
this  goal  are: 

•  Pursue  the  development  and  deployment  of  digital  management  information 
systems. 
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•  Ensure  that  Navy  Medical  communication  systems  are  integrated  to  store  and 
transfer  medical  information  throughout  the  Department  of  Defense 

•  Encourage  the  use  of  appropriate  available  technologies  to  enhance  and 
measure  quality  of  care,  readiness,  patient  satisfaction,  resource  utilization,  and 
efficiency. 

•  Obtain  and  distribute  current  hardware  and  software  and  train  Medical 
Department  personnel  for  collection  of  accurate  and  timely  information. 


The  most  applicable  strategy  to  this  thesis  is  the  third,  which  defines  the  objectives  of 
new  technologies  and  identifies  five  factors  to  be  used  in  creation  of  metrics.  Although  it 
is  left  up  to  local  organizations  to  develop  systems  of  measuring  progress  toward  these 
goals,  six  measures  are  identified  that  are  linked  to  one  or  more  of  these  objectives  as 
follows.  (See  Figure  2) 


Figure  2.  Steps  to  identify  MOEs 


B.  MOE  IDENTIFICATION 

Figure  3  shows  the  six  proposed  MOEs  that  when  used  together,  can  satisfy  all  the 
strategic  goals  defined  in  section  IV.A. 
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Strategic  Goal 

Quality  of  Care 


MOEs 

Outcomes 


Return  to  Duty  Times 

Number  of  Medevacs 

Patient  Satisfaction 
Surveys 

Physician  Satisfaction 
Survey 

Utilization  Rate 
Figure  3.  Cross-Mapping  of  MOEs  to  Strategic  Goals 

1.  Outcome  Measures  (quality  of  care) 

The  objective  in  outcome  management  is  to  optimize  quality.  As  discussed 
previously,  quality  is  a  foundation  for  acceptance  and  adoption  of  new  clinical  processes. 
There  is  currently  no  standard  tool  in  place  to  measure  changes  in  health  care  outcomes 
due  to  telemedicine.  Furthermore,  personnel  in  quality  assurance  and/or  performance 
improvement  positions  have  no  vehicle  to  identify  whether  the  outcome  of  a  telemedicine 
consult  is  different  when  telemedicine  is  used.  Although  some  sites  are  formulating 
protocols  to  be  used  for  telemedicine  consults,  none  have  been  identified  that  consistently 
and  primarily  address  outcomes.  Outcomes  measurement  can  be  done  however,  on  an 
audit  basis  through  the  use  of  a  short  form  that  directly  asks  the  physician  if  the 
telemedicine  consult  has  resulted  in  a  a  change  of  diagnosis,  disposition,  or  treatment.  If 
negative  outcomes  cannot  be  associated  with  the  use  of  telemedicine  (or  positive 
outcomes  can)  it  suggests  that  telemedicine  is  being  effective. 
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2.  Return  to  Duty  Times  (readiness) 

In  DoD,  the  time  span  between  time  of  injury  and  return  to  full  duty  is  of 
particular  interest,  since  the  time  spent  by  an  active  duty  member  receiving  healthcare  is 
time  lost  in  performing  his/her  primary  duties.  In  other  words,  the  member  is  not 
available  and  readiness  is  reduced.  Managers  are  interested  in  maximizing  the  time  that  a 
member  is  in  full  duty  status.  Thus,  the  benefit  of  telemedicine  can  be  measured  in  terms 
of  decreased  time  that  it  takes  to  travel  to  the  treatment  facility,  receive  treatment,  and 
return.  Depending  on  the  duty  location  and  the  location  of  the  medical  facility,  this  could 
result  in  several  hours  or  several  weeks.  Also,  depending  on  the  rank  and  duties  of  the 
patient,  the  manpower  and  monetary  costs  could  vary  significantly.  In  a  thesis  done  by 
Craigmiles  (1995),  the  travel  time  of  a  typical  consult  was  2  hours  for  a  round  trip  and  the 
typical  patient  was  an  E-l  in  training.  However,  the  average  number  of  consults  per  year 
is  20,000  which  results  in  a  potential  training  time  savings  of  40,000  hours.  In  contrast, 
in  Bosnia,  the  round  trip  travel  time  was  weeks,  typically  resulting  in  a  medical 
evacuation  to  Germany,  and  the  typical  patient  was  a  higher  enlisted  rate  providing 
operational  expertise.  The  number  of  consults  was  low,  at  39  over  two  and  a  half  months. 
Using  an  average  of  2  weeks  lost  for  each  medevac,  this  results  in  a  total  time  savings  of 
13,104  hours.  In  both  cases,  this  implies  that  readiness  was  improved  since  the  patient 
was  returned  to  full  duty  status  quickly.  Return  to  duty  times  were  improved  but 
comparing  these  times  as  discussed  earlier  can  be  misleading,  since  the  definition  of 
readiness  is  different  in  a  variety  of  military  settings  and  has  yet  to  be  defined  with  a 
standard  set  of  indicators.  Studies  continue  to  use  the  pay  grade  of  the  service  member  to 
calculate  a  costs  savings,  but  this  is  exceptionally  subjective  by  the  rater  and  nearly 
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impossible  to  standardize.  What  is  more  certain  is  the  fact  that  successful  telemedicine 
consults  occurred  which  resulted  in  no  need  for  travel. 

On  the  surface,  return  to  duty  times  could  be  a  useful  measurement  to  contribute 
to  the  level  of  readiness,  however  the  validity  is  lost  in  the  analysis  of  various  scenarios 
that  could  prove  to  discount  the  measure.  Specifically,  telemedicine  may  be  a  reason  for 
increasing  the  time  before  return  to  duty.  Reliability,  defined  as  the  degree  to  which  the 
measurement  is  free  of  error,  is  considered  a  necessary  but  insufficient  condition  for 
validity  (Benbasat  &  Moore,  1992).  In  the  case  of  return  to  duty  times,  it  is  difficult  to 
estimate  the  level  of  error  and  whether  the  MOE  would  be  overestimated  or 
underestimated.  The  types  of  errors  possible  are  illustrated  in  scenarios  such  as  persons 
on  limited  duty  but  still  being  capable  of  performing  their  primary  jobs.  Furthermore,  by 
identifying  a  condition,  telemedicine  can  create  a  limited  duty  and  prevent  a  casualty  or 
medical  retirement.  These  situations  are  easily  lost  and  therefore  not  reported,  or  only 
partially  reported.  Thus  effectiveness  of  telemedicine  can  not  be  determined  by 
measuring  return  to  duty  times. 

3.  Number  of  Medical  Evacuations  (Medevacs)  (readiness) 

The  largest  and  most  complicated  medical  transportation  system  is  operated  by 
the  DoD  Military  Health  Services  System.  Economies  of  scale  have  forced  the  DoD  to 
consolidate  and  specialize  its  hospitals  and  clinics.  Consequently,  specialized  medical 
care  cannot  be  provided  at  every  installation.  With  personnel  located  at  many,  widely- 
dispersed  DoD  installations,  the  DoD  is  forced  to  transport  a  patient  to  the  proper  facility. 
It  is  proposed  that  telemedicine  offers  highly  increased  readiness  and  significant  savings 
by  preventing  a  number  of  medevacs.  When  a  patient  is  in  a  remote  area  and  the 
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attending  physician  does  not  have  the  resources  to  make  an  adequate  diagnosis,  the 
correct  disposition  defaults  to  a  medevac.  Of  course,  medevacs  are  expensive  due  to  the 
cost  of  the  flight,  TAD  costs,  and  the  cost  of  losing  the  worker.  Medevacs  also  include 
risks  if  the  patient  is  in  a  combat  zone  or  at  sea.  When  patients  are  at  sea,  they  must  be 
transported  by  helicopter,  or  by  “high-lining”  patients  between  ships,  where  a  cable  is 
literally  strung  between  the  ships  and,  using  pulleys,  the  patient  is  pulled  to  the  other 
vessel.  The  strategic  mission  is  delayed  until  this  procedure  has  been  completed  and  the 
member  is  on  his  way  to  safety.  In  its  most  simple  form,  telemedicine  must  be  assumed 
to  be  effective  if  it  causes  the  number  of  medevacs  to  be  reduced. 

4.  Patient  Satisfaction  Measurements  (patient  satisfaction) 

An  easier  measurement  .that  can  be  standardized  in  a  one-size-fits-all  manner,  is  a 
patient  satisfaction  survey,  but  results  already  published  suggest  that  most  patients  find 
telemedicine  an  acceptable  way  to  receive  medical  care  (Grigsby,  1996).  In  a  thesis  done 
by  Craigmiles  (1995),  satisfaction  measurements  of  52  patients  within  one  month  were 
measured  and  it  was  found  that  the  telemedicine  system  neither  positively  nor  negatively 
impacted  the  patients’  perception  of  staff  attitude,  the  time  spent  in  discussing  the 
patients’  medical  problems,  or  the  overall  perception  of  care.  This  is  consistent  with 
Grigsby’s  theory  that  telemedicine  has  been  generally  accepted  in  a  traditional  setting. 

Although  patient  satisfaction  remains  a  principal  approach  to  identifying  problems 
within  the  overall  health  care  system,  it  loses  its  validity  concerning  telemedicine  due  to 
biases  incurred  in  the  attractiveness  of  a  patient  embarking  on  an  incentive  trip.  Other 
factors  can  discredit  patient  satisfaction  surveys,  especially  in  the  military  where  patients 
are  stationed  in  remote  areas  and  would  be  better  suited  to  receive  health  care  in  a  more 
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populated  area.  Moreover,  patients  typically  desire  movement  to  a  more  populated  area. 
Patients  must  be  allowed  to  provide  feedback  into  the  system  by  assessing  their 
satisfaction  with  the  entire  process.  However,  if  they  know  that  this  process  will  prevent 
them  from  being  moved,  results  may  be  skewed  since  there  are  personal  benefits  to  be 
realized  by  a  medevac  aside  from  the  military  mission.  Morale  is  extremely  important. 

In  remote  areas,  members  are  typically  under  extreme  pressure  to  perform  under  stressful 
conditions.  Liberty  and  leave  are  granted  under  the  military’s  purview  and  an 
opportunity  for  medical  liberty  or  convalescent  leave  is  attractive  and  sometimes  the 
commanding  officer  will  opt  for  the  medical  evacuation  to  take  advantage  of  the  possible 
rejuvenation  of  morale.  This  factor  has  been  suggested  as  attributable  to  minimal  use  of 
telemedicine  in  Bosnia,  since  commanding  officers  do  not  want  to  be  dealing  with 
possible  long-term  care  by  either  telemedicine  or  traditional  care.  Commanding  Officers 
believe  that  the  best  place  to  receive  care  is  outside  of  a  stressful  environment,  and  that 
the  the  member  should  be  evacuated.  Thus,  there  is  a  bias  in  the  patient  survey  construct 
and  it  should  only  be  used  in  stable  environments,  not  for  operational  environments. 

5.  Physician  Satisfaction  Measurements  (efficiency,  resource  utilization) 

The  objective  in  this  factor  is  to  provide  a  high  technology  environment  as  desired 
and/or  needed  by  health  care  providers.  The  focus  of  many  technologies  is  to  improve 
efficiency  and  resource  utilization--it  does  what  it  is  supposed  to  do  and  it  is  being  used. 

It  is  hardly  surprising  that  satisfaction  of  the  healthcare  provider  is  often  overlooked. 
Often  the  hype  of  technology  or  patient  dissatisfaction  encompasses  telemedicine  projects 
and  the  attitudes  of  the  health  care  provider  is  not  assessed.  So  when  the  equipment  is 
not  used,  the  information  systems  analyst  is  puzzled.  It  is  true  that  addressing  and 
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resolving  technical  issues  is  a  vital  part  of  developing  a  successful  telemedicine  program. 
However,  debates  over  what  is  adequate  resolution  quality,  compression  ratios,  whose 
codec  is  better,  what  is  adequate  bandwidth,  and  the  plethora  of  other  technical  issues 
represent  in  part  a  denial  on  the  part  of  industry  experts  to  grapple  with  the  truly  thorny 
issues  facing  telemedicine.  It  needs  to  be  identified  why  or  why  not  physicians  choose 
telemedicine.  An  attitudes  assessment  will  identify  many  issues  that  are  not  apparent  to 
management  personnel  and  provide  the  manager  with  benchmarks  for  improvement. 
Generally,  if  the  health  care  providers  accept  and  express  satisfaction  with  what 
telemedicine  is  doing  and  express  their  intentions  to  use  telemedicine,  it  can  be  assumed 
that  there  is  a  favorable  impact  on  the  effectiveness  of  telemedicine. 

6.  Utilization  Rate  (resource  utilization) 

There  is  an  objective  to  decrease  the  time  required  of  specialists  (Greenauer, 

1995)  and  optimize  the  time  giving  care  by  specialists.  This  is  a  measure  of  the  level  at 
which  telemedicine  consults  compare  with  traditional  consults.  By  accounting  for  the 
number  of  telemedicine  consults  performed  and  received,  it  can  be  compared  to  the 
number  of  traditional  consults  in  the  same  specialty.  This  number  can  also  be  compared 
with  the  survey  data  asking  the  health  care  provider  what  he  anticipates  his  workload  to 
be.  In  this  way,  there  is  a  goal  to  be  achieved,  and  although  the  goal  may  change,  at  least 
there  is  a  method  for  knowing  how  we  are  moving  toward  or  from  the  goal.  The 
utilization  rate  is  certainly  not  a  measure  of  how  well  the  equipment  is  working.  It  is  a 
reflection  of  the  performance  aggregate  spanning  the  entire  spectrum  of  care,  affected  by 
other  influences  such  as  management  practices,  clinical  protocols,  education,  and 
dialogue  necessary  to  accomplish  the  mission  of  the  facility.  If  telemedicine  is  being 
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used  consistently  or  increasingly,  it  can  lead  to  the  understanding  that  telemedicine  is 
being  effective. 

C.  PROPOSED  SET  OF  MOE'S 

In  summary,  it  is  recommended  that  physician  satisfaction  measurements, 
utilization  rates,  outcomes  measures,  and  number  of  medevacs  be  used  as  the  foundations 
for  measures  of  effectiveness.  Although  this  thesis  does  not  go  in  depth  to  analyze  their 
validity,  their  merits  initially  warrant  use  in  this  study.  Next  it  becomes  desirable  to 
collect  the  data  needed  for  these  MOE’s  in  a  practical  manner. 
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V.  CASE  STUDY:  IMPLEMENTATION  OF  TELEMEDICINE  AT  ROTA 

NAVAL  HOSPITAL,  SPAIN 

A.  BACKGROUND 

The  primary  role  of  the  Naval  Station,  Rota,  Spain  is  to  support  the  U.S.  Sixth 
Fleet  operating  in  the  Mediterranean.  While  the  Sixth  Fleet  tackles  the  increasingly 
complex  task  of  promoting  peace  and  stability  in  the  Mediterranean  area,  the  Naval 
Station  Rota  and  its  tenant  commands  maintain  support  to  the  operating  forces.  Naval 
Hospital,  Rota  is  a  forward  deployed  Medical  Treatment  Facility  serving  military 
members  and  their  family  members,  retired  members  and  their  family  members,  and 
DOD  contracted  civilians  stationed  in  Rota  Spain.  Areas  of  response  include  the  Iberian 
Peninsula,  Western  Mediterranean,  and  North  Africa  while  also  providing  support  to  the 
Sixth  Fleet,  DOD  and  NASA  shore  activities,  and  primary  surgical  team  response  for  the 
European  theater.  Other  hospitals  in  the  area  are  Naval  Hospitals  Naples  (with  satellite 
clinics  at  Gaeta,  LaMaddelena,  Pintamare,  and  Capodichino)  and  Sigonella  (with  a  clinic 
at  Souda  Bay). 

The  head  of  the  Management  Information  Systems  Department  (MISD)  at  Rota 
has  been  present  for  the  full  sequence  of  events  of  the  telemedicine  deployment  at  Rota 
and  has  been  accepted  by  the  command  as  the  primary  and  credible  source  of  information 
on  use  of  telemedicine.  He  has  remained  proactive  and  informed  on  the  progress  of 
technology  outside  of  Rota  and  has  managed  resources  to  remain  integrated  with  DoD. 
During  a  site  visit  in  April  1997,  it  was  routinely  asked  of  healthcare  providers,  “What 
would  you  recommend  to  be  the  next  step  in  the  use  of  telemedicine  at  Rota?  Where 
would  you  invest  resources?”  The  typical  answer  was  “I’m  not  sure.  I’d  go  to  MISD.” 
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There  are  two  clinical  champions  primarily  responsible  for  encouraging  clinical  use.  One 
is  the  flight  surgeon,  a  computer  “superuser”  who  encourages  healthcare  providers  to  look 
outside  of  Rota  for  a  network  of  solutions.  The  other  is  the  dermatologist,  the  primary  • 
user,  who  is  continuing  to  realize  the  benefits  of  off-site  consultation  and  sees  the 
advantages  of  using  telemedicine  to  avoid  moving  the  patient.  Another  advocate  is  the 
Commanding  Officer,  a  neurosurgeon  dedicated  to  encouraging  use  of  telemedicine  as  he 
markets  its  successes  and  potential  uses.  His  views  on  telemedicine  are  known 
throughout  the  command  and  most  employees  state  that  the  successes  at  Rota  are  largely 
due  to  his  support  of  the  program.  He  does  not  believe  that  this  new  process  warrants 
extra  resources,  however,  and  has  resisted  committing  extra  resources  at  this  time.  He 
emphasizes  that  telemedicine  is  a  change  in  current  processes  and  should  not  be  confused 
with  a  new  process.  In  summary.  Rota  is  populated  with  clinical  champions,  an 
information  systems  advocate,  and  a  central  authority  figure  all  in  tune  with  driving  the 
success  of  telemedicine. 

The  Naval  Medical  Information  Management  Center  (NMIMC)  took  the  lead  and 
deployed  telemedicine  to  the  European  theater  in  September,  1996,  funding  the 
implementation  costs.  Included  in  the  deployment  was  delivery  of  dermatology  scopes, 
ENT  scopes,  and  VTC  units.  There  was  no  local  plan  or  needs  assessment  prior  to 
delivery  and  installation  of  the  equipment;  however,  the  sites  were  given  ownership  and 
entire  control  of  the  program.  The  deployment  consisted  of  a  site  visit  by  NMIMC 
personnel  who  configured  and  tested  the  equipment  and  trained  local  personnel.  While 
the  equipment  was  installed,  NMIMC  personnel  held  a  two-day  conference  with  key  local 
players  to  discuss  issues  as  follows: 
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•  Customer  expectations 

•  Role  of  NMIMC  Detachment  in  supporting  existing  and  new  infrastructure 

•  Working  knowledge  of  the  effect  of  telemedicine  and  other  technology  on 
overseas  sites 

•  Ambulatory  care  data  collection  and  business  process  reengineering  efforts 

•  Reemphasis  to  the  COs  and  CIOs  the  significance  of  the  LCM  and  funding 
changes  as  part  of  a  long  term  strategic  approach 

Collection  of  metrics  was  discussed,  and  although  it  was  agreed  that  it  was  important  and 
necessary,  there  was  not  a  consensus  on  what  metrics  to  gather.  At  a  later  conference, 
outcomes  measures  was  committed  as  the  highest  priority  of  measurements.  There  had 
been  no  prior  metrics  gathered  to  use  as  a  comparison  and  a  baseline.  The  fleet  surgeon 
at  CINCUSNAVEUR  is  insistent  that  new  technologies  should  not  be  implemented 
without  a  valid  business  case  analysis  prior  to  deployment.  That  is  what  occurred  in  this 
case,  however,  and  it  is  now  necessary  to  begin  to  study  the  effects  of  telemedicine  at  the 
local  level  without  the  benefit  of  a  baseline. 

Although  Rota  received  an  ENT  scope  and  a  dermatology  scope  as  part  of  the 
initial  setup,  these  pieces  of  equipment  were  later  shipped  to  satellite  clinics  while  Rota 
retained  the  VTC  and  ordered  new  scopes  as  replacements.  This  was  done  in  order  to 
give  more  remote  areas  access  to  a  medical  treatment  facility.  The  VTC  was  placed  in 
the  emergency  room  in  order  to  provide  access  by  medical  personnel  and  also  to 
emphasize  ownership  by  healthcare  providers. 

At  a  meeting  held  in  Naples  on  June  4, 1996,  Rota  had  identified  four  areas  as. 
consultation  capabilities  to  offer  surrounding  sites:  dermatology,  ENT,  urology,  and 
radiology.  Shortly  after,  it  was  concurred  by  representatives  CINCUSNAVEUR,  the 
MTF’s  and  NMIMC  that  a  dermatology  study  would  be  initiated  between  Naples  and 
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Rota.  Sigonella,  Naples,  LaMadellena,  Gaeta,  and  Souda  Bay  received  derm  scopes  and 
the  dermatologist  at  Rota  prepared  to  receive  consults. 

B.  DATA  COLLECTION  MODEL 

It  is  proposed  to  use  a  four-area  model  to  collect  telemedicine  data  using  surveys 
of  health  care  providers,  outcomes  measurements,  site  profiles,  and  the  utilization  rate. 
Each  area  can  affect  the  others.  Additional  information  can  be  gleaned  by  examining 
them  together.  For  example,  if  outcome  measurements  identify  a  positive  change  in  how 
care  is  provided,  the  physicians  would  reflect  that  in  a  satisfaction  survey  and  the  usage 
of  the  tool  should  increase.  The  usage  rate  can  then  be  displayed  in  the  sites  profile. 

Thus  the  relationships  are  described  as  causal  and  an  inconsistency  between  any  of  the 
areas  can  identify  why  there  is  a  positive/negative  utilization. 

The  following  four-area  model  illustrates  the  flow  and  influence  of  data  collection 
across  the  spectrum  of  health  care.  The  model  identifies  two  levels  of  relationships 
among  the  entities.  A  bold  line  represents  a  strong  relationship  with  direct  influence  and 
the  dotted  line  represents  a  more  subtle  but  identifiable  influence.  The  function  of  the 
surveys  is  to  collect  physician  satisfaction  and  utilization  data  particular  to  telemedicine. 
The  function  of  outcomes  surveys  is  to  collect  changes  in  outcomes  and  number  of 
medevacs  avoided.  The  profile  information  is  data  displayed  to  support  marketing,  and 
utilization  data  is  collected  and  displayed  to  provide  a  view  of  how  much  telemedicine  is 
being  used--the  bottom  line. 
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Survey 


Outcomes 


Figure  4.  Proposed  Data  Collection  Model— Influences 


1.  Health  Care  Provider  Surveys 

As  a  measure  of  physician  satisfaction  and  intent  to  use,  it  is  important  to 
emphasize  that  before  health  care  providers  are  quizzed  on  the  many  aspects  of 
telemedicine  it  is  vital  to  educate  them  on  the  subject  to  avoid  ignorance  and  confusion. 
Understanding  the  questions  will  provide  a  much  better  answer  from  health  care 
providers  on  an  assessment  than  if  they  had  never  been  exposed  to  the  concept.  The 
following  questions  were  asked,  and  other  questions  follow  that  can  be  used.  It  is 
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especially  beneficial  to  perform  the  assessment  with  a  face-to-face  interview,  if  possible. 
This  has  several  advantages.  First,  it  is  not  necessary  to  rely  on  the  physician  to  complete 
the  survey  at  their  leisure  and  return  it,  which  would  most  likely  result  in  a  poor  response 
rate.  Second,  it  is  beneficial  to  place  emphasis  on  the  importance  of  establishing  a 
personal  connection  with  each  of  the  interviewees,  ensuring  the  interviews  were 
interactive  and  collaborative  in  nature.  This  results  in  a  building  of  trust  among 
information  systems  managers  and  health  care  providers  and  also  allows  interpretation  of 
non-verbal  language.  The  focus  of  this  survey  is  to  determine  what  has  worked  and  what 
has  not  worked  for  the  provider  in  the  implementation  of  telemedicine.  This  is  especially 
interesting  because  it  looks  inside  the  heads  of  physicians. 

Demographics:  age,  years  in  the  Navy,  months  at  present  location,  job 
specialty,  years  in  job  specialty,  rank.  These  are  collected  for  comparison  and  possible 
trends.  It  is  sometimes  easier  to  understand  responses  if  the  assessor  has  a  documented 
mental  profile  of  the  respondee,  and  is  especially  useful  if  a  face-to-face  interview  is  not 
possible. 

Has  your  use  of  telemedicine  been  successful?  Why  or  why  not?  This  is 
oriented  toward  the  user  who  has  been  using  telemedicine  long  enough  to  make  a 
conclusion  about  the  program’s  success  in  general.  Kotter  (1995)  identifies  eight  major 
errors  made  by  organizations  during  a  planned  change: 

•  Not  establishing  a  great  enough  sense  of  urgency 

•  Not  creating  a  powerful  enough  guiding  coalition 

•  Lacking  a  vision 

•  Undercommunicating  the  vision  by  a  factor  of  ten 

•  Not  removing  obstacles  to  the  new  vision 

•  Not  systematically  planning  for  and  creating  short-term  wins 
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•  Declaring  victory  too  soon 

•  Not  anchoring  changes  in  the  corporation’s  culture 

This  question  attempts  to  identify  reasons  why  the  organization  as  a  whole  is  having 
difficulty  implementing  a  planned  change.  If  these  options  are  listed,  the  question 
encourages  the  respondee  to  give  an  honest,  big  picture,  “bottom  line”  and  then  to 
provide  further  comments  to  support  it.  Of  course  there  can  be  many  reasons  why 
changes  fail  and  Kotter  continues  to  state  that  even  successful  change  efforts  are  “messy 
and  full  of  surprises.”  But  a  vision  of  the  change  process  can  reduce  the  error  rate,  and 
fewer  errors  can  spell  the  difference  between  success  and  failure. 

Have  there  been  positive/negative  impacts  on  how  you  practice  medicine  (i.e. 
time  required,  quality  assurance)?  Describe.  One  of  the  problems  with  implementing 
a  new  process  is  the  impression  that  there  will  be  more  to  do  or  the  current  process  will 
become  more  complex.  This  question  attempts  to  find  out  if  telemedicine  is  making 
healthcare  easier  or  more  difficult  in  the  eyes  of  the  healthcare  provider. 

Do  you  feel  you  are  proficient  with  information  technology?  Lack  of 
experience  or  training  has  often  been  a  barrier  to  implementation  of  any  information 
system.  A  new  piece  of  equipment  is  often  met  with  the  fear  of  the  unknown  and  fear  of 
change.  Acknowledging  this  is  half  the  battle,  but  once  they  connect  learning  a  new 
discipline  to  solving  an  important  problem,  there  is  momentum  to  acclimate  with  the 
technology.  In  addition,  interfaces  are  still  being  designed  that  are  much  better  at 
bridging  the  jump  from  machine  to  user.  A  bad  interface  will  cause  adults  to  be  “turned 
off’  and  claim  to  be  computer  illiterate  (Negroponte,  1996). 

Do  you  feel  that  there  has  been  adequate  management  support  at  (1)  your 
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site,  (2)  in  the  Department  of  the  Navy,  (3)  In  the  Department  of  Defense,  (4)  in  the 
private  sector?  This  question  is  centered  towards  bridging  the  gap  between  clinical 
personnel  and  management  across  organizational  levels.  Often  the  burden  of  a 
unsuccessful  implementation  is  shifted  to  an  external  entity  and  the  fundamental  problem 
is  no  longer  in  the  local  scope.  This  question  attempts  to  determine  if  there  is  a 
deficiency  with  management  support,  and  more  importantly,  where  they  perceive  that 
deficiency  to  be. 

Do  you  feel  that  there  has  been  adequate  clinical  support  at  (1)  your  site,  (2) 
in  the  Department  of  the  Navy,  (3)  In  the  Department  of  Defense,  (4)  in  the  private 
sector?  This  addresses  the  same  issue  as  the  previous  question,  but  the  subject  is  now 
within  the  healthcare  provider’ s-  culture. 

Do  you  feel  outcomes  in  the  form  of  diagnoses,  treatment,  or  dispositions  of 
the  patient  have  improved  as  a  result  of  telemedicine  use?  Give  an  example.  This 
provides  a  general  answer  to  the  question  of  whether  or  not  there  are  changes  in  clinical 
outcomes.  This  can  provide  useful  information  in  the  case  that  a  specific  outcomes 
management  protocol  is  not  in  place  or  under  evaluation.  It  could  also  prove  useful  to 
compare  it  to  specific  outcomes  data  gathered  through  a  clinical  outcomes  data  input 
sheet  for  each  telemedicine  consult. 

Which  types  of  telemedicine  do  you  use?  (with  listing)  The  listing  of 
capabilities  changes  very  quickly  and  the  health  care  provider  may  not  be  aware  of  what 
can  be  provided  with  telemedicine.  So  not  only  does  this  identify  what  is  currently 
occurring  at  the  site,  it  subtly  keeps  the  clinician  more  up-to-date  with  technologies  that 
could  be  used. 
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The  following  questions  are  those  useful  in  cases  where  it  is  yet  to  be  determined 
which  applications  to  consider  for  telemedicine  implementation.  In  a  needs  assessment 
done  by  Grigsby  for  the  University  of  Colorado  Hospital  (O’Connor,  1997),  attitude 
assessments  are  broken  down  into  attitudes  toward  the  hub  facility  and  attitudes  toward 
telemedicine.  A  primary  determinant  for  a  telemedicine  program’s  success  is  the 
willingness  of  the  remote  site  to  engage  in  a  cooperative,  collaborative  relationship  with 
the  tertiary  hub  site.  A  lack  of  willingness  may  result  from  a  variety  of  factors,  but  can  be 
narrowed  down  to  two  general  components:  (1)  ill  feelings  towards  the  tertiary  hub  site 
stemming  from  negative  past  experiences;  and  (2)  the  remote  health  care  providers 
feeling  threatened  by  the  presence  provided  via  telemedicine  by  the  hub  facility.  The 
following  questions  are  extremely  open-ended  and  it  is  relatively  necessary  to  conduct  the 
assessment  in  a  face-to-face  meeting. 

What  is  your  general  impression  of  the  hub  facility?  This  is  a  question  of  how 
highly  the  hub  facility  is  regarded  by  the  local  facility.  The  existing  organizational 
dynamics  may  be  negatively  influencing  the  potential  willingness  to  collaborate  and 
engage  in  clinical  dialogue.  If  there  are  preconceptions  concerning  the  success  of 
collaboration  or  a  doubt  concerning  the  level  of  hub  expertise,  the  rewards  for  initiating  a 
telemedicine  consult  may  be  preconceived  to  be  non-existent. 

Do  you  have  any  negative  attitudes  towards  the  hub  facility?  If  so,  why? 

This  question  may  bring  out  specific  cases  that  have  given  the  remote  facility  cause  to 
avoid  confrontation  with  the  hub  facility.  Negative  outcomes,  whether  they  be  business- 
oriented  or  human-oriented  have  a  definite  impact  on  willingness  to  collaborate. 

However,  problems  identified  with  this  question  are  typically  symptoms  of  a  larger 
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problem  not  yet  identified,  or  simply  a  misunderstanding  that  was  never  resolved. 

Do  you  have  any  positive  attitudes  toward  the  hub  facility?  If  so,  why? 
Again,  this  would  bring  out  reasons  to  pursue  telemedicine  and  identify  critical  success 
factors  that  can  be  capitalized  by  the  remote  sites,  or  even  duplicated  by  other  systems. 

Do  you  have  concerns  about  how  a  telemedicine  relationship  with  the  hub 
facility  may  affect  you  or  your  practice?  No  system  is  ever  completely  free  from 
potential  problems  and  brainstorming  the  “what  if  s”  could  prevent  negative  encounters 
or  provide  scenarios  for  backup  plans. 

What  is  your  familiarity  with  telemedicine?  As  with  the  prior  assessment  form, 
this  identifies  whether  or  not  the  health  care  provider  is  aware  of  the  capabilities  and 
limitations  of  telemedicine.  It  is  helpful  to  know  if  the  healthcare  provider  has  had 
hands-on  training,  or  has  done  extensive  scholarly  research  on  the  use  of  telemedicine.  It 
may  also  help  to  identify  potential  clinical  champions. 

Do  you  have  any  concerns  about  the  use  of  telemedicine?  This  is  the  catch-all 
question  regarding  the  technological  or  managerial  issues  that  can  change  almost  daily. 

What  is  your  perception  of  the  possible  advantages  and  disadvantages  of 
telemedicine?  This  is  more  of  a  “faith”  question  regarding  telemedicine  and  may 
provide  answers  in  response  to  the  education  received  thus  far.  Health  care  providers  will 
be  determining  whether  the  possible  advantages  become  true  advantages  or  evolve  into 
disadvantages. 

Would  telemedicine  benefit  your  practice?  Telemedicine  is  not  suited  for  all 
specialties,  i.e.  those  specialties  requiring  smell  or  touch  and  less  cognitive  in  nature. 
However,  there  may  be  applications  that  have  never  been  tried  but  have  potential  benefits. 
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What  is  the  likelihood  of  you  using  telemedicine?  Even  if  it  has  been  identified 
that  use  of  telemedicine  should  result  in  benefits,  there  may  be  underlying  reasons  why 
the  physicians  will  not  use  it.  If  the  physician  is  not  yet  ready  for  the  change,  it  would  be 
better  serve  the  site  to  wait  until  obstacles  have  been  removed  before  proceeding. 

How  often  would  you  use  it?  In  anticipation  for  implementation,  a  baseline 
penetration  rate  provides  a  goal  to  achieve.  If  measured  and  monitored  correctly,  the 
target  is  a  benchmark  of  success  and  a  way  to  anticipate  whether  the  application  will  be 
cost  beneficial.  It  has  also  been  found  that  participation  increases  perceived  system 
quality,  which  in  turn  increases  involvement.  Involvement  then  influences  both  the 
attitudes  towards  the  system  and  system  usage.  These  attitudes  together  with  the 
subjective  norm  are  hypothesized  to  influence  one’s  intention  to  use  the  system  (Bensabat 
&  Moore,  1992). 

Would  you  actively  or  passively  support  the  development  of  a 
telemedicine/distance  learning  program  with  other  sites?  Important  to  any  new 
implementation  is  the  role  of  the  clinical  champion.  Taking  the  lead  and  becoming 
involved  in  their  circle  of  peers  can  make  or  break  a  new  implementation.  If  these 
champions  can  be  identified  early,  momentum  can  be  maintained  and  a  healthy  balance  of 
experts  will  sustain  the  program. 

2.  Changes  in  Outcomes 

The  following  questions  were  included  in  a  form  used  during  telemedicine 
consults  with  the  purpose  of  collecting  changes  in  outcomes  and  medevac  information.  It 
must  be  used  consistently  for  a  specified  amount  of  time  in  order  to  provide  an  unbiased 
measure  in  which  all  consults  are  used  and  not  just  favored  cases  or  environments.  These 
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measures  can  then  be  placed  appropriately  within  the  site  profile  to  confirm  the 
application  at  the  site  when  it  is  felt  that  the  application  is  acceptable  to  be  used  for 
resource  sharing.  It  is  also  important  that  the  health  care  provider  realize  that  the 
measurement  of  outcomes  is  not  concerned  with  physicians  doing  a  “better”  job.  It  is 
concerned  with  physicians  being  more  certain  of  the  outcome  and  having  the  tools  to 
provide  a  more  timely  treatment.  In  addition  to  outcomes,  medevac  information  has  been 
embedded  as  questions  to  support  measures  of  reductions  in  medevacs. 

Date,  length  of  consult,  patient  FMP/SSN.  The  date  is  recorded  for  tracking 
purposes  and  as  a  way  to  uniquely  identify  the  consult.  The  image,  or  other  supporting 
digital  information,  is  stored  using  a  locally  devised  but  uniform  combination  of  the 
patient’s  last  name,  family  member  prefix,  social  security  number,  and  date  as  an 
identifying  stamp.  The  length  of  consult  can  also  be  used  to  glean  information  on  how 
long  telemedicine  consults  are  taking  and  to  give  the  provider  appropriate  workload 
credit.  The  Clinical  Health  Care  System  {CHCS)  requires  a  patient  to  be  registered  prior 
to  receiving  care.  Since  this  is  not  achieved  quickly  and  easily  by  the  physician,  the 
physician  does  not  use  CHCS  and  the  manual  record  remains  the  easiest  way  to  document 
that  the  provider  is  spending  clinical  time  providing  telemedicine  consults. 

Permanent  duty  station,  age,  and  service  of  patient.  This  is  essential  in 
providing  patient  demographics  to  be  used  in  strategically  and  methodically  locating 
health  care  providers.  Since  there  may  only  be  one  specialist  in  the  geographical  area,  it 
would  be  wise  to  locate  him/her  centrally— closest  to  where  most  of  the  patients  are  or 
where  they  are  anticipated  to  be  based  on  past  trends.  In  addition,  this  information  in 
conjunction  with  the  service  of  the  patient  and  whether  they  are  active  duty,  can  provide 
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data  that  may  prove  useful  in  performing  a  cost  savings  study. 

ICD-9-CM  (International  Classification  of  Diseases,  Ninth  Revision,  Clinical 
Modification).  This  is  a  classification  system  that  groups  related  disease  entities  and 
procedures  for  the  reporting  of  statistical  information.  The  clinical  modification  of  the 
ICD-9  was  developed  by  the  National  Center  for  Health  Statistics  for  use  in  the  United 
States.  The  use  of  ICD-9-CM  promotes  uniformity  of  health  care  data  by  establishing 
coding  and  classification  standards  and  guidelines  across  health  care  settings.  It  is  the 
core  of  the  information  infrastructure.  The  ICD-9-CM  must  be  as  accurate  as  possible  in 
order  to  arrive  at  a  binding  clinical  decision  and  be  able  to  validate  the  continuum  of  care. 

Is  the  image  acceptable  to  be  used  as  a  determining  factor  in  your  consult? 
This  is  an  often  overlooked  question  that  plays  an  important  part  in  a  measure  of  success. 
If  the  image  (or  other  digital  object)  is  not  of  diagnostic  quality  according  to  the  provider, 
it  must  still  be  known  that  the  encounter  occurred.  Telemedicine  consults  that  are  not 
usable  are  opportunities  for  technical  improvement.  Only  the  specialist  is  qualified  to 
make  a  diagnostic  decision,  and  he/she  is  the  one  to  be  held  responsible  for  the  decision 
to  use  the  consult. 

Image  ID#.  This  should  be  used  for  definite  tracking  and  can  be  used  to  verify 
that  the  correct  digital  object  is  associated  with  the  patient’s  record. 

Is  a  medevac  recommended  based  on  this  consult?  To  where?  This  is  directly 
related  to  a  measure  in  medevac  avoidance.  In  most  cases,  if  a  telemedicine  consult 
cannot  be  performed,  the  medevac  is  initiated.  If  the  telemedicine  consult  is  performed 
and  the  physician  is  not  comfortable  making  a  diagnosis  based  on  the  image,  it  is  most 
certainly  appropriate  to  initiate  a  medevac.  Another  scenario  exists  where,  based  on  the 
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telemedicine  consult,  the  specialist  wishes  to  continue  care  in  person.  The  piece  of 
information  most  useful  to  the  medevac  MOE  is  the  number  of  times  that  a  medevac  is  no 
longer  required  based  on  a  telemedicine  consult.  It  represents  a  direct  savings  of  time  and 
money  and  is  a  “bottom  line”  to  provide  as  useful  information  to  the  local  commander 
regarding  the  cost  of  running  a  practice,  and  also  supports  assessments  and  conclusions 
when  compared  to  the  trend  of  aggregate  medevacs.  If  a  medevac  IS  recommended  it  is 
also  useful  to  know  where  the  patient  is  being  sent,  again,  to  provide  demographic  data 
and  support  the  appropriate  location  of  specialists. 

Is  a  non-medical  attendant  requested?  In  approximately  half  of  medevac  cases 
overseas,  a  non-medical  attendant  is  approved  to  accompany  the  patient  to  the  consult 
site.  This  is  mostly  due  to  possible  traveling  difficulties  due  to  the  foreign  customs, 
languages,  and  cultures  that  may  prove  uncomfortable  to  patients.  Again,  this  provides 
information  that  can  be  used  in  cost  savings  studies. 

Recommended  treatment  and  disposition.  This  is  standard  information  for 
consults  and  documents  the  appropriateness  of  the  health  care  provided. 

Based  on  this  consult,  has  the  diagnosis/treatment/disposition  changed?  This 
is  the  key  piece  of  information  useful  to  outcomes  research.  The  specialist  performing 
the  consult  alone  knows  if  there  could  have  been  a  different  outcome  without  the  use  of 
telemedicine.  Furthermore,  each  application  is  different  and  any  combination  of  the  three 
may  have  incurred  a  change.  The  specialist  is  also  cognizant  that  the  use  of  telemedicine 
is  making  a  difference  and  the  question  may  provide  a  positive  reinforcement  and  further 
use. 
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3.  Site  Profiles 


The  objective  of  this  set  of  questions  is  to  provide  an  up-to-date  source  of 
information  in  order  to  support  the  network  necessary  for  medical  collaboration.  The 
Department  of  Defense  medical  network  of  providers  spans  the  globe.  Currently,  in 
Europe,  the  sites  have  developed  an  informal,  undocumented  method  to  determine  health 
care  needs  in  the  area  of  response.  However,  there  are  many  pieces  of  information  that 
are  difficult  to  obtain  and  often  that  information  changes  depending  on  who  is  asked.  The 
Defense  Medical  Regulating  Information  System  (DMRIS)  is  not  providing  an  accurate 
comprehensive  list  of  medical  capabilities  although  it  has  the  capability  of  doing  so. 
Furthermore,  the  physicians  do  not  necessarily  have  access  to  DMRIS;  thus,  when  they 
need  additional  consultation,  they  find  the  outside  source  themselves  and  arrange  for  an 
accepting  physician  for  a  medevac.  DMRIS  has  no  requirement  for  accommodating  the 
needs  of  telemedicine— it  serves  as  a  medical  regulating  function  by  facilitating  requests 
for  patient  transportation.  Patient  regulation  is  what  telemedicine  is  trying  to  avoid. 
Optimally,  the  system  supporting  telemedicine  should  provide  quick  and  reliable  access 
to  a  profile  of  each  specialty  and  sites  with  telemedicine  capabilities. 

Profiles  can  be  developed  and  advertised  to  not  only  the  area  of  response  but  to 
the  rest  of  the  Department  of  Defense  for  collaboration  at  any  time  and  anywhere.  As  a 
result  of  the  information  gathered  from  the  surveys  and  clinical  outcomes,  a  profile  can 
be  generated  to  provide  that  information  outside  of  the  command  for  further  exploitation. 
At  a  minimum,  the  following  should  be  part  of  a  telemedicine  profile:  facility  type,  store 
and  forward  vs.  VTC  capabilities  and  standards  compliance  for  specific  specialties, 
physicians  available  for  specific  specialties,  bandwidth  available,  and  use  of  the  WWW. 
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Facility  type.  This  identifies  whether  the  treatment  facility  was  a  clinic,  hospital, 
or  teaching  hospital.  It  should  also  include  a  telephone  number  for  general  information 

Store  and  forward  vs.  VTC  capabilities  and  standards  compliance  for 
specific  specialties.  It  is  necessary  to  know  whether  the  technology  is  compatible 
between  the  sending  and  receiving  sites.  Store  and  forward  would  require  a  compatible 
file  attachment  protocol  like  binhex,  MIME,  uudecoded,  or  FTP.  In  certain  specialties 
like  digital  radiology,  it  is  also  necessary  to  know  if  the  compression  format  is  standard, 
such  as  with  the  DICOM3  standard.  The  VTC  also  must  be  compatible  with  standards 
such  as  H.320. 

Physicians  available  for  specific  specialties.  As  discussed  before,  specialized 
medical  care  cannot  be  provided  at  every  installation,  and  it  is  necessary  for  the  patient  to 
have  access  to  these  specialists  and  for  administrators  to  know  where  these  specialties 
are.  The  telemedicine  profile  should  identify  those  health  care  providers  that  are 
proficient  with  the  use  of  telemedicine  and  it  should  identify  a  way  to  contact  them  for  a 
consult.  This  can  be  expanded  quite  extensively.  Some  sites  have  developed  a  web- 
based  physician  profile  for  viewing  which  describes  the  physician’s  practice. 

Bandwidth  available.  This  question  is  necessary  in  order  to  anticipate  the 
resolution  of  an  interactive  VTC  consult.  Sometimes  all  that  is  needed  is  for  a  physician 
to  direct  attention  to  a  general  area  by  pointing  in  order  to  emphasize  the  problem  to  a 
remote  site.  In  this  case,  a  128KBPS  may  be  adequate.  In  other  cases,  however,  it  might 
be  desirable  to  see  an  ultrasound  with  enough  speed  and  resolution  to  diagnose  and  a  T1 
connection,  may  be  required. 

Use  of  the  World  Wide  Web.  Currently  in  Navy  medicine,  this  is  a  very 
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inexpensive  way  to  gather  and  disseminate  information.  As  a  result  of  a  CHCS 
telecommunications  requirement,  Navy  medical  treatment  facilities  are  all  connected  or 
have  access  to  a  global  Wide  Area  Network  (WAN)  with  a  minimum  rate  of  56KBPS  and 
most  sites  have  constructed  a  home  page.  Pictures,  the  site  profiles,  a  feedback  page,  and 
even  an  electronic  consult  can  easily  be  constructed  and  viewed  by  any  computer 
connected  to  the  network.  Part  of  the  profile  would  be  a  “home  page”  to  go  to  for 
information  on  the  site’s  telemedicine  capabilities,  objectives,  measures  of  effectiveness, 
and  results  of  surveys.  The  barrier  to  accomplishing  this  is  simply  lack  of  education 
and/or  initial  staffing. 

4.  Utilization  Rate 

Currently  this  is  being  gathered  manually  by  physicians  providing  the 
telemedicine  consult.  Since  consults  are  still  few  in  numbers,  it  is  relatively  easy  to 
document  how  many  consults  have  occurred  and  how  long  they  took  to  complete.  Within 
CHCS,  there  is  an  option  to  track  “teleconsults.”  This  was  originally  designed  with 
telephone  consults  from  patients  in  mind  and  not  for  tracking  or  transferring  images  and 
consulting  by  VTC.  Knowing  how  much  the  technology  is  being  used  is  important  since 
it  is  the  bottom  line  for  any  information  system.  It  may  be  the  most  efficient,  elegant 
system  in  the  developer’s  mind,  yet  it  is  not  being  used.  This  serves  to  identify  that  there 
is  a  fault  occurring  in  the  overall  system  and  all  prior  efforts  will  be  for  naught  unless  the 
fault  is  identified  and  corrected.  Utilization  is  the  most  important  and  revealing  measure 
of  telemedicine.  It  has  been  suggested  that  the  real  benefit  to  be  realized  by  telemedicine 
is  that  of  increased  use  (Edwards,  1995),  which  he  calls  a  “second  order  benefit”  and  is 
the  goal  of  any  new  system. 
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C.  APPLICATION  TO  THE  DATA  COLLECTION  MODEL 

Using  the  data  collection  model  previously  described,  a  case  study  is  presented 
using  the  environment  at  the  Naval  Hospital  Rota. 

1.  Health  Care  Provider  Surveys 

Two  days  prior  to  my  site  visit  to  Rota,  a  short  survey  was  distributed  to 
healthcare  providers  and  information  systems  personnel.  Upon  arrival,  I  was  greeted 
throughout  the  chain  of  command  and  given  an  opportunity  to  speak  at  several  of  the 
clinical  meetings  held  in  the  command.  During  these  meetings  I  informed  listeners  that  I 
was  there  to  gather  information  across  all  functionalities  to  include  attitudes  and  to 
identify  where  telemedicine  was  used  or  desired.  I  was  also  granted  time  to  speak  to 
several  physicians,  nurses,  and  hospital  corpsmen.  I  distributed  the  same  survey  to  Naval 
Hospitals  Sigonella  and  Naples  remotely  as  pilot  surveys;  however  the  response  rate  was 
significantly  lower  at  around  34%  and  17%,  respectively.  I  attributed  this  to  lack  of 
visibility  and,  from  what  I  was  told,  an  overwhelming  number  of  surveys  being 
distributed  within  the  last  year.  Furthermore,  the  previous  surveys  had  not  reported 
results  back  to  the  command  and  there  was  not  an  incentive  to  provide  survey  information 
based  on  the  merits  of  the  survey  alone. 

There  are  36  doctors  and  dentists  stationed  at  Rota,  and  23  responded  to  the 
survey  for  a  response  rate  of  64%.  When  comparing  this  rate  to  that  at  Sigonella  and 
Naples,  it  is  easy  to  see  how  visibility  and  top-level  support  can  make  a  difference  in 
participation.  The  statistical  data,  which  may  prove  useful  in  later  studies,  resulted  in  the 
following: 
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•  The  average  age  of  the  respondee:  40. 

•  The  average  time  served  in  the  Navy:  1 5  years. 

•  The  average  time  at  Rota:  16  months. 

•  The  average  years  in  specialty:  9  years. 

The  results  of  the  survey  are  as  shown  in  the  following  table: 


Has  your  use  of  telemedicine  been  successful? 

8  yes 

4  no 

Have  there  been  positive/negative  impacts  on  how  you  practice 
medicine? 

8  positive 

1  negative 

Do  you  feel  you  are  proficient  with  information  technology? 

14  yes 

9  no 

Do  you  feel  that  there  has  been  adequate  management  support 
at  your  site? 

16  yes 

2  no 

in  DoN? 

1 1  yes 

5  no 

in  DoD? 

9  yes 

5  no 

in  the  private  sector? 

7  yes 

2  no 

Do  you  feel  there  has  been  adequate  clinical  support  at  your 

15  yes 

3  no 

site? 

in  DoN? 

16  yes 

3  no 

in  DoD? 

9  yes 

3  no 

in  the  private  sector? 

6  yes 

2  no 

Do  you  feel  outcomes  in  the  form  of  diagnoses,  treatment,  or 
dispositions  of  the  patient  have  improved  as  a  result  of 
telemedicine  use? 

11  yes 

1  no 

There  was  additional  space  for  each  question  to  provide  further  explanation  and  rationale 
for  the  responses.  This  proved  especially  helpful  in  interpreting  non-responses  and 
negative  responses  for  specific  questions.  For  example,  the  radiologist  stated  that  he  did 
not  believe  telemedicine  was  successful  at  Rota  because  they  did  not  have  the  appropriate 
equipment  for  true  digital  radiology.  It  is  also  noteworthy  that  respondees  mostly  felt  that 
Rota  was  doing  better  than  other  sites  and  civilian  counterparts.  Although  valid 
conclusions  about  Rota  cannot  be  made  based  on  the  small  number  of  respondees,  the 
general  feeling  is  positive  in  terms  of  what  has  been  done  locally  and  generally  with 
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telemedicine. 


2.  Changes  in  Outcomes 

Two  doctors  assisted  in  constructing  a  data  collection  form  to  be  used  for 
identifying  changes  in  outcomes  for  telemedicine  consults.  From  the  period  of  Nov  96 
through  Jul  97,  using  this  form,  the  dermatologist,  identified  the  following: 

•  There  were  7  changes  in  diagnosis.  Upon  his  observation  of  the  image  the 
initial  diagnosis  of  the  remote  consultation  changed. 

•  There  were  9  changes  in  treatment.  The  initial  treatment  changed,  possibly 
related  to  the  change  in  diagnosis. 

•  There  were  1 1  changes  in  disposition.  Again,  this  change  in  the  patients  status 
makes  a  direct  impact  on  the  readiness  of  the  active  duty  member. 

•  Of  the  14  consults,  12  were  considered  acceptable  images  for  a  consultation. 
The  other  two  were  considered  a  poor  picture,  or  of  inadequate  resolution. 

•  There  were  9  consults  identified  as  “non-medevacs”.  In  other  words,  if  there 
had  not  been  a  telemedicine  consult,  these  cases  would  have  been  medevaced  for 
observation  elsewhere. 

Again,  although  statistically  binding  conclusions  cannot  be  made,  there  is  an 
implication  of  favorable  outcomes  and  a  definite  avoidance  of  medevacs.  This  provides 
the  site  with  a  basis  for  evaluating  telemedicine  as  being  effective. 

3.  Site  Profiles 

Currently,  the  network  of  outside  consultation  within  the  DoD  network  is 
accomplished  by  informal,  undocumented  methods.  In  short,  there  is  not  an  acceptable 
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standardized  system  of  providing  a  visible  profile  usable  for  the  function  of  telemedicine. 
At  a  minimum,  needed  information  is  facility  type,  store  and  forward  or  VTC  capabilities 
and  standards  compliance  for  specific  specialties,  physicians  available  for  specific 
specialties,  bandwidth  available,  and  use  of  the  WWW.  For  example,  the  Rota  profile  at 
a  minimum,  is  found  to  be: 

Rota:  Naval  Hospital  01 1-3456-82-3673,  DSN  727-3673 
S&F  tmedrota@med.navy.mil 
VTC  011-3456-82-5555,  PictureTel,  H.320 
Dermatology  -  Dr.  Pidkowicz  01 1-3456-82-5555 
ISDN  -  256KBPS 
http://192.101.124.40/ 

4.  Utilization  Rate 

The  utilization  rate  reflects  the  level  of  use  in  comparison  with  traditional 
methods.  At  Rota,  the  number  of  telemedicine  consults  were  almost  entirely  attributed  to 
dermatology.  During  the  observation  period  (November  96  to  Jul  97),  there  were  14 
dermatology  consults.  There  were  approximately  400  dermatology  consults  during  the 
same  time  period  for  a  aggregated  utilization  rate  of  3.5%.  This  leads  to  questions 
regarding  possible  cases  that  were  not  presented  for  use  by  telemedicine.  For  a  short 
time,  one  of  the  satellite  hubs  contracted  the  services  of  a  local  dermatologist  for  cases 
that  could  have  been  serviced  by  telemedicine.  This  leads  to  an  assumption  that  there  is 
still  sufficient  skepticism  by  many  healthcare  providers.  There  has  been  minimal 
education  which  ultimately  will  negatively  affect  use  of  any  system.  Since  there  have 
been  no  negative  experiences  with  telemedicine  noted  or  comparative  data  (the  Rota 
dermatologist  is  the  sole  Navy  provider  in  Europe),  the  utilization  rate  can  be 
benchmarked  as  an  introductory  measure  and  monitored  for  change.  The  only  other 


57 


consults  were  2  pediatric  VTC  consultations,  and  other  specialties  such  as  radiology  have 
yet  to  be  explored  for  use,  with  the  understanding  that  the  specialties  must  not  be 
aggregated  into  one  because  of  the  variant  suitability  of  telemedicine  to  different 
functionalities.  Regardless  of  the  result,  it  is  necessary  to  document  utilization  rates  and 
observe  trends  in  order  to  identify  potential  investment  areas. 

D.  GAP  ANALYSIS:  NEEDS  ASSESSMENT 

A  needs  assessment  addresses  the  attitudes  of  the  physicians  and  this  is  where  a 
physician  survey  can  prove  invaluable.  Additional  questions  can  be  framed  to  obtain 
information  regarding  the  skills  and  interests  of  the  health  care  providers,  how  busy  their 
practices  were,  and  to  gather  their  opinions  on  the  health  care  needs  of  the  community. 
Consequently,  this  involves  a  more  comprehensive  and  subjective  survey  than  the  one 
distributed  to  the  physicians  at  Rota.  A  general  needs  assessment  was  not  done  at  Rota, 
thus  doing  a  satisfaction  survey  may  create  confusion.  This  is  illustrated  by  eight  surveys 
that  were  entirely  blank  except  for  a  comments  that  it  was  not  used  or  they  didn't  know 
what  it  was.  The  same  number  of  surveys  were  complete  and  descriptive,  with  the 
remaining  surveys  being  complete  and  brief.  This  illustrates  an  inconsistent  level  of 
knowledge  among  users  at  Rota.  This  is  a  gap  in  the  system  identified  with  the  use  of  the 
surveys  and  this  lack  of  education  can  be  considered  to  be  a  contributing  factor  to  the 
low  penetration  rate. 

When  it  is  rationalized  that  it  is  unethical  to  consider  costs  and  thought  of 
qualitatively,  telemedicine  is  always  considered  appropriate  if  it  MIGHT  have  benefit. 
Using  the  utilization  rate  can  support  requirements  for  telemedicine.  Although  health 
care  providers  at  Rota  believe  telemedicine  has  benefit  and  outcomes  are  favorable  (i.e. 
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consults  are  identifying  favorable  changes  in  outcomes),  they  are  using  telemedicine  at 
what  appears  to  be  a  low  rate  based  on  twice  the  rate  of  dermatology  consults  that 
occurred  in  Bosnia.  This  also  suggest  a  resistance  to  the  change  in  business  practices, 
and  that  significant  training,  education,  and  dialogue  needs  to  occur  before  telemedicine 
can  provide  optimal  benefits. 

It  has  been  recognized  (O’Connor,  1996)  (Allen,  1996)  that  telemedicine 
technology  hype  and  its  potential  to  cure  some  of  what  ails  the  health  care  system  results 
in  a  scramble  to  implement  telemedicine  networks.  This  rush  is  at  the  expense  of  a 
rational,  objective  process  to  determine  whether  a  telemedicine  program  is  actually 
needed,  let  alone  welcome.  After  administration  and  analysis  of  a  needs  assessment,  a 
strategic  plan  can  be  developed  outlining  the  needed  services  as  well  as  the  most 
appropriate  telemedicine  technology  to  provide  those  services.  A  needs  assessment  after 
implementation  should  not  answer  the  question,  “Did  we  do  the  right  thing?”  It  should 
answer  the  question,  “What  should  we  do?” 

In  summary,  looking  at  the  qualitative  measures,  physician  satisfaction  is 
favorable,  outcomes  are  favorable,  medevacs  have  been  avoided,  but  utilization  is  low 
which  suggests  a  resistance  to  change.  Telemedicine  should  be  integrated  into  the 
training  and  education  of  all  personnel  and  its  success  should  be  marketed  outside  the 
command  through  the  use  of  the  World  Wide  Web. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Medical  decisions  are  certainly  complex,  highly  variable,  and  poorly  supported  • 
with  clinical  studies.  This  is  a  difficult  foundation  on  which  to  place  a  telemedicine 
program.  But  there  is  a  push  by  DoD  physicians  and  managers  to  be  a  testbed  and 
identify  successes  and  failures.  This  data  is  essential  to  prove  the  effectiveness  of 
telemedicine.  The  methodology  shown  in  this  thesis  is  a  proposed  beginning  to  establish 
and  maintain  a  system  for  measuring  the  effectiveness  of  telemedicine.  Measurements  of 
effectiveness  (physician  satisfaction,  outcome  measures,  medevacs  avoided,  and 
utilization)  can  support  telemedicine.  Based  on  evaluation  of  these  measures,  a  gap 
analysis  can  identify  where  resources  should  be  directed. 

B.  RECOMMENDATIONS 

While  traditional  consults  are  documented  in  the  organic  Composite  Health  Care 
System  (CHCS),  a  telemedicine  consult  is  not,  since  the  patient  is  not  registered  outside 
the  organic  CHCS.  Thus,  methods  for  gathering  data  remains  a  manual  process,  i.e. 
tedious  and  error  prone.  The  methodology  stated  here  is  dependent  upon  the  commitment 
of  everyone  in  the  process  to  contribute  to  data  collection.  It  would  increase  the 
reliability  of  outcomes  and  utilization  data  to  ensure  that  all  telemedicine  consults  are 
captured  by  CHCS  for  further  use  by  planners. 

It  is  the  responsibility  of  each  local  facility  to  maintain  an  appropriate  set  of 
measures  to  support  locally  defined  objectives.  Each  facility  has  a  unique  strategic 
mission  leading  todifferent  objectives  to  be  achieved.  Furthermore,  measurement  is  an 
iterative  process,  requiring  successive  refinement  from  one  year  to  the  next  and  a 
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program  must  be  in  place  to  ensure  this.  It  is  important  that  the  program  be  driven  from 
the  top  down  and  top  officials  should  understand  the  weaknesses  in  measures  of 
effectiveness,  and  use  them  strictly  as  indicators  rather  than  absolutes.  They  should  be 
kept  simple  so  that  they  are  understood,  meaningful,  and  useful. 

It  is  a  mistake  to  think  that  further  deployment  of  new  technology  will  succeed 
without  the  involvement  or  "buy-in"  of  the  health  care  providers.  Taking  the  time  to  do 
face-to-face  interviews  and  engaging  in  productive  dialogue  between  managers  and 
healthcare  providers  can  produce  much  more  information  than  by  second-guessing  and 
filling  out  forms. 

Needs  assessments  must  be  timely  and  iterative.  Ex  post  facto  studies  have  little 
value  in  a  planned  change  where  it  is  necessary  for  users  to  be  actively  involved  in  the 
change  while  it  is  happening,  not  after.  Needs  assessments  should  be  focused  on  the 
strategic  goals  and  objectives  before  implementation  of  the  technology. 

C.  AREAS  FOR  FURTHER  STUDY 

Outcomes  management  is  an  area  that  merits  further  study,  especially  when 
observing  the  various  changes  that  may  occur  due  to  a  telemedicine  consultation; 
specifically,  changes  in  diagnosis,  treatment,  and  disposition.  The  interaction  of  one 
change  to  another  is  a  complex  situation  that  may  prove  to  be  of  immense  value  in  the 
study  of  outcomes.  A  clinical  data  set  with  an  adequate  number  of  data  elements  must  be 
collected  and  categorized  by  specialty  to  support  this.  Outcomes  management  is  of 
increasing  interest  with  managed  care  specialists,  and  the  changes  in  types  of  care  are 
especially  of  interest  when  examining  impact  on  the  healthcare  system. 

Further  effort  should  be  spent  on  a  more  in-depth  analysis  of  the  validity  of  the 
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proposed  MOE's,  possibly  through  the  use  of  a  Delphi  group. 

Additional  research  is  also  necessary  in  the  development  of  a  useful  needs 
assessment  to  include  physician  surveys  as  new  telemedicine  methods  become  available. 
This  should  focus  on  creating  the  right  mix  of  questions  to  be  asked  in  order  to  provide 
the  most  useful  feedback. 
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